Sep 07 10:09:31.485725 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 07 10:09:31.485738 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 07 10:09:31.485746 ip-10-0-132-168 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 07 10:09:31.486060 ip-10-0-132-168 systemd[1]: Failed to start Kubernetes Kubelet. Sep 07 10:09:41.485141 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 07 10:09:41.485162 ip-10-0-132-168 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 11d8a42be5d9400ebe3b63aa2089b711 -- Sep 07 10:11:16.257235 ip-10-0-132-168 systemd[1]: Starting Kubernetes Kubelet... Sep 07 10:11:16.672859 ip-10-0-132-168 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 10:11:16.672859 ip-10-0-132-168 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 07 10:11:16.672859 ip-10-0-132-168 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 10:11:16.672859 ip-10-0-132-168 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 07 10:11:16.672859 ip-10-0-132-168 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 10:11:16.675396 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.675318 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 07 10:11:16.688762 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688740 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:11:16.688762 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688759 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:11:16.688762 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688763 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:11:16.688762 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688766 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688769 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688772 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688775 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688777 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688780 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688784 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688789 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688793 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688797 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688800 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688803 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688806 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688809 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688812 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688815 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688818 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688820 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688823 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:11:16.688921 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688826 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688828 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688831 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688833 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688835 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688838 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688841 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688843 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688846 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688848 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688851 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688853 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688856 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688858 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688861 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688864 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688867 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688870 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688872 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688875 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:11:16.689393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688877 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688880 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688883 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688885 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688887 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688890 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688892 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688896 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688907 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688910 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688912 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688915 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688917 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688920 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688923 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688925 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688949 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688953 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688955 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688958 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:11:16.689867 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688962 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688965 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688967 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688970 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688972 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688975 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688977 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688981 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688984 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688986 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688989 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688991 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688994 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688996 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.688999 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689002 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689004 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689007 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689010 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689012 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:11:16.690393 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689016 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689020 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689023 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689025 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689028 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689031 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689033 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689035 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689040 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689043 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689047 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689050 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689052 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689056 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689058 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689061 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689064 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689752 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689758 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:11:16.690871 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689762 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689765 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689767 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689770 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689773 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689775 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689778 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689780 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689783 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689786 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689789 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689791 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689794 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689796 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689798 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689802 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689805 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689807 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689810 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689812 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:11:16.691382 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689815 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689818 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689820 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689823 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689825 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689828 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689830 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689833 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689836 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689838 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689840 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689843 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689846 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689848 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689851 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689853 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689856 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689860 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689864 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:11:16.691882 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689867 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689870 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689873 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689876 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689879 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689882 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689884 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689887 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689890 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689897 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689899 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689902 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689905 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689908 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689911 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689915 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689919 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689922 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689925 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:11:16.692368 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689950 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689955 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689959 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689962 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689965 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689967 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689970 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689972 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689975 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689977 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689980 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689982 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689985 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689987 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689989 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689992 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689994 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.689997 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690000 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690002 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:11:16.692825 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690005 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690007 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690010 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690013 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690016 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690018 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690021 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690025 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690028 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690031 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690033 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690036 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690039 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690041 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690044 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690047 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690049 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690052 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690054 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690426 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:11:16.693329 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690431 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690433 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690436 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690439 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690442 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690444 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690447 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690449 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690452 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690454 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690457 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690459 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690462 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690464 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690467 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690470 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690473 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690475 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690478 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690481 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:11:16.693805 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690483 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690486 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690489 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690493 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690496 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690499 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690502 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690505 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690508 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690511 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690513 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690516 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690519 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690522 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690524 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690527 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690530 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690532 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690535 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:11:16.694294 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690537 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690547 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690549 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690552 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690554 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690557 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690559 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690562 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690564 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690567 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690570 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690573 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690575 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690578 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690581 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690583 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690586 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690588 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690590 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:11:16.694748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690593 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690596 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690600 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690602 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690605 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690607 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690609 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690612 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690614 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690617 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690619 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690621 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690624 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690626 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690629 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690631 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690633 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690636 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690639 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690642 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:11:16.695256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690645 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690647 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690650 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690654 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690657 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690659 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690662 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690664 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690666 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690670 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690674 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690677 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690679 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690682 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690685 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690687 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690690 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690693 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690695 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.690698 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:11:16.695758 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690776 2565 flags.go:64] FLAG: --address="0.0.0.0" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690818 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690825 2565 flags.go:64] FLAG: --anonymous-auth="true" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690830 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690834 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690837 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690841 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690845 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690848 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690851 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690856 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690860 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690863 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690865 2565 flags.go:64] FLAG: --cgroup-root="" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690868 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690871 2565 flags.go:64] FLAG: --client-ca-file="" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690875 2565 flags.go:64] FLAG: --cloud-provider="external" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690878 2565 flags.go:64] FLAG: --cluster-dns="[]" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690885 2565 flags.go:64] FLAG: --cluster-domain="" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690888 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690892 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690895 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690898 2565 flags.go:64] FLAG: --container-log-max-files="5" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690902 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 07 10:11:16.696341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690905 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690908 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690911 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690914 2565 flags.go:64] FLAG: --contention-profiling="false" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690917 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690919 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690923 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690926 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690949 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690952 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690955 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690958 2565 flags.go:64] FLAG: --enable-load-reader="false" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690960 2565 flags.go:64] FLAG: --enable-server="true" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690963 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690972 2565 flags.go:64] FLAG: --event-burst="100" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690975 2565 flags.go:64] FLAG: --event-qps="50" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690978 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690981 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690985 2565 flags.go:64] FLAG: --eviction-hard="" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690988 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690991 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690994 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690997 2565 flags.go:64] FLAG: --eviction-soft="" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.690999 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691002 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 07 10:11:16.696939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691006 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691009 2565 flags.go:64] FLAG: --experimental-mounter-path="" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691012 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691014 2565 flags.go:64] FLAG: --fail-swap-on="true" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691017 2565 flags.go:64] FLAG: --feature-gates="" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691020 2565 flags.go:64] FLAG: --file-check-frequency="20s" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691024 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691027 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691029 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691032 2565 flags.go:64] FLAG: --healthz-port="10248" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691035 2565 flags.go:64] FLAG: --help="false" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691038 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-132-168.ec2.internal" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691041 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691044 2565 flags.go:64] FLAG: --http-check-frequency="20s" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691046 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691049 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691052 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691055 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691058 2565 flags.go:64] FLAG: --image-service-endpoint="" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691061 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691063 2565 flags.go:64] FLAG: --kube-api-burst="100" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691066 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691069 2565 flags.go:64] FLAG: --kube-api-qps="50" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691072 2565 flags.go:64] FLAG: --kube-reserved="" Sep 07 10:11:16.697559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691074 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691078 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691081 2565 flags.go:64] FLAG: --kubelet-cgroups="" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691084 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691087 2565 flags.go:64] FLAG: --lock-file="" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691090 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691092 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691095 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691100 2565 flags.go:64] FLAG: --log-json-split-stream="false" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691104 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691106 2565 flags.go:64] FLAG: --log-text-split-stream="false" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691109 2565 flags.go:64] FLAG: --logging-format="text" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691111 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691115 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691117 2565 flags.go:64] FLAG: --manifest-url="" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691120 2565 flags.go:64] FLAG: --manifest-url-header="" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691124 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691127 2565 flags.go:64] FLAG: --max-open-files="1000000" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691131 2565 flags.go:64] FLAG: --max-pods="110" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691134 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691140 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691143 2565 flags.go:64] FLAG: --memory-manager-policy="None" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691146 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691149 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691152 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 07 10:11:16.698194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691154 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691168 2565 flags.go:64] FLAG: --node-status-max-images="50" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691171 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691174 2565 flags.go:64] FLAG: --oom-score-adj="-999" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691176 2565 flags.go:64] FLAG: --pod-cidr="" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691179 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691184 2565 flags.go:64] FLAG: --pod-manifest-path="" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691186 2565 flags.go:64] FLAG: --pod-max-pids="-1" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691191 2565 flags.go:64] FLAG: --pods-per-core="0" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691193 2565 flags.go:64] FLAG: --port="10250" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691196 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691200 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06e150e41d2c2e403" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691203 2565 flags.go:64] FLAG: --qos-reserved="" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691206 2565 flags.go:64] FLAG: --read-only-port="10255" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691209 2565 flags.go:64] FLAG: --register-node="true" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691212 2565 flags.go:64] FLAG: --register-with-taints="" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691219 2565 flags.go:64] FLAG: --registry-burst="10" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691223 2565 flags.go:64] FLAG: --registry-qps="5" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691226 2565 flags.go:64] FLAG: --reserved-cpus="" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691228 2565 flags.go:64] FLAG: --reserved-memory="" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691232 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691235 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691237 2565 flags.go:64] FLAG: --rotate-certificates="false" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691240 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691243 2565 flags.go:64] FLAG: --runonce="false" Sep 07 10:11:16.698834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691245 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691249 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691252 2565 flags.go:64] FLAG: --seccomp-default="false" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691255 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691257 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691260 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691263 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691266 2565 flags.go:64] FLAG: --storage-driver-password="root" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691269 2565 flags.go:64] FLAG: --storage-driver-secure="false" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691271 2565 flags.go:64] FLAG: --storage-driver-table="stats" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691274 2565 flags.go:64] FLAG: --storage-driver-user="root" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691277 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691279 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691282 2565 flags.go:64] FLAG: --system-cgroups="" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691285 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691291 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691294 2565 flags.go:64] FLAG: --tls-cert-file="" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691296 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691302 2565 flags.go:64] FLAG: --tls-min-version="" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691304 2565 flags.go:64] FLAG: --tls-private-key-file="" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691307 2565 flags.go:64] FLAG: --topology-manager-policy="none" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691318 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691321 2565 flags.go:64] FLAG: --topology-manager-scope="container" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691324 2565 flags.go:64] FLAG: --v="2" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691328 2565 flags.go:64] FLAG: --version="false" Sep 07 10:11:16.699460 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691333 2565 flags.go:64] FLAG: --vmodule="" Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691362 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.691366 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691481 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691485 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691488 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691491 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691494 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691497 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691499 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691503 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691505 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691508 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691511 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691513 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691515 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691518 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691521 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691523 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691526 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:11:16.700083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691528 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691530 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691534 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691537 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691539 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691543 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691545 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691548 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691550 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691554 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691557 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691560 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691562 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691566 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691568 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691571 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691574 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691576 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691579 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:11:16.700565 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691581 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691584 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691586 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691589 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691591 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691594 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691596 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691599 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691601 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691604 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691606 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691609 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691611 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691614 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691616 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691620 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691622 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691624 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691628 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691630 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:11:16.701043 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691633 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691635 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691638 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691640 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691643 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691645 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691648 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691651 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691653 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691656 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691658 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691660 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691663 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691666 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691668 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691671 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691673 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691675 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691679 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691682 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:11:16.701541 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691684 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691687 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691689 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691692 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691694 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691697 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691699 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691704 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691706 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691709 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691713 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691715 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691718 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691720 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691723 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691725 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691728 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691730 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691734 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691737 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:11:16.702099 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691740 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691743 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.691746 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.692483 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.700439 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.700454 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.701355 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.701362 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701425 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701431 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701435 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701438 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701441 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701444 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701447 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:11:16.702621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701450 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701453 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701455 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701458 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701460 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701464 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701466 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701469 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701472 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701474 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701477 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701479 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701482 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701484 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701487 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701489 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701492 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701494 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701497 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:11:16.703026 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701499 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701502 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701504 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701514 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701517 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701520 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701523 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701525 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701528 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701531 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701534 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701536 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701539 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701541 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701544 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701547 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701549 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701553 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701555 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701558 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:11:16.703499 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701560 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701563 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701565 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701568 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701570 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701573 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701575 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701578 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701580 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701582 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701585 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701587 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701590 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701592 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701595 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701598 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701608 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701612 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701615 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701617 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:11:16.704000 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701620 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701622 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701624 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701627 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701630 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701632 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701634 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701637 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701640 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701642 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701644 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701647 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701649 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701652 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701654 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701658 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701661 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701664 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701667 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:11:16.704473 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701670 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701673 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701676 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701678 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701681 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701683 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701685 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701688 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701690 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701693 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701701 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701704 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701707 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701710 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.701716 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 10:11:16.704965 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701877 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701882 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701885 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701888 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701891 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701894 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701897 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701899 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701902 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701904 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701907 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701909 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701912 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701914 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701917 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701919 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701922 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701925 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701939 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701942 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:11:16.705320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701944 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701947 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701949 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701952 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701954 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701957 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701959 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701962 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701972 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701974 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701977 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701979 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701982 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701984 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701987 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701991 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701995 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.701997 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702000 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:11:16.705798 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702003 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702006 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702009 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702011 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702014 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702016 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702019 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702022 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702024 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702027 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702030 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702032 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702035 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702038 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702040 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702042 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702045 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702048 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702050 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702052 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:11:16.706274 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702055 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702057 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702066 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702069 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702071 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702074 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702076 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702080 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702083 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702086 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702088 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702091 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702093 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702095 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702098 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702100 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702103 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702106 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702109 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:11:16.706749 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702113 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702116 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702119 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702121 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702123 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702126 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702129 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702131 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702133 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702136 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702138 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702141 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702143 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702146 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702148 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702151 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702159 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702161 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702163 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702166 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:11:16.707233 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:16.702168 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:11:16.707760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.702173 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 10:11:16.707760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.702285 2565 server.go:962] "Client rotation is on, will bootstrap in background" Sep 07 10:11:16.707760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.704746 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 07 10:11:16.707760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.705963 2565 server.go:1019] "Starting client certificate rotation" Sep 07 10:11:16.707760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.706075 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 10:11:16.707760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.706950 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 10:11:16.708832 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.708819 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 10:11:16.716083 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.716063 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 10:11:16.727140 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.727123 2565 log.go:25] "Validated CRI v1 runtime API" Sep 07 10:11:16.730440 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.730421 2565 log.go:25] "Validated CRI v1 image API" Sep 07 10:11:16.731245 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.731233 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 07 10:11:16.733781 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.733763 2565 fs.go:135] Filesystem UUIDs: map[42d19bda-e82d-4ef1-abaf-dd811f20e4f9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f0ac6d15-bb03-431b-8639-e32e4e805601:/dev/nvme0n1p3] Sep 07 10:11:16.733840 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.733781 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 07 10:11:16.739133 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.739017 2565 manager.go:216] Machine: {Timestamp:2026-09-07 10:11:16.737572967 +0000 UTC m=+0.378081784 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100499 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214c395c2f56b0d6417673720126bb SystemUUID:ec214c39-5c2f-56b0-d641-7673720126bb BootID:11d8a42b-e5d9-400e-be3b-63aa2089b711 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:51:63:ad:3f:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:51:63:ad:3f:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:43:90:1b:70:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 07 10:11:16.739133 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.739125 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 07 10:11:16.739280 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.739200 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 07 10:11:16.739519 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.739503 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 10:11:16.742816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.742789 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 07 10:11:16.742969 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.742818 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 07 10:11:16.743749 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.743739 2565 topology_manager.go:138] "Creating topology manager with none policy" Sep 07 10:11:16.743784 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.743752 2565 container_manager_linux.go:309] "Creating device plugin manager" Sep 07 10:11:16.743784 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.743759 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 10:11:16.744602 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.744592 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 10:11:16.746469 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.746459 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 07 10:11:16.750844 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.750833 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 07 10:11:16.750980 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.750971 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 07 10:11:16.751031 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.751001 2565 kubelet.go:486] "Attempting to sync node with API server" Sep 07 10:11:16.751031 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.751017 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 07 10:11:16.751031 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.751030 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 07 10:11:16.751155 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.751039 2565 kubelet.go:398] "Adding apiserver pod source" Sep 07 10:11:16.751155 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.751047 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 07 10:11:16.751841 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.751400 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 10:11:16.751898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.751870 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 10:11:16.756028 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.756011 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 07 10:11:16.761022 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761002 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 07 10:11:16.761111 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761050 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 07 10:11:16.761111 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.761087 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 07 10:11:16.761212 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.761131 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 07 10:11:16.761212 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761210 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 07 10:11:16.761273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761222 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 07 10:11:16.761273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761228 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 07 10:11:16.761273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761233 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 07 10:11:16.761273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761241 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 07 10:11:16.761273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761247 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 07 10:11:16.761877 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761868 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 07 10:11:16.761908 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761878 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 07 10:11:16.761908 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761885 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 07 10:11:16.761908 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761891 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 07 10:11:16.761908 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761905 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 07 10:11:16.762053 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.761914 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 07 10:11:16.762789 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.762780 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 07 10:11:16.762789 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.762789 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 07 10:11:16.764438 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.764421 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 07 10:11:16.766399 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.766386 2565 server.go:1268] "Started kubelet" Sep 07 10:11:16.766485 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.766455 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 07 10:11:16.766532 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.766476 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 07 10:11:16.766586 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.766532 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 07 10:11:16.767256 ip-10-0-132-168 systemd[1]: Started Kubernetes Kubelet. Sep 07 10:11:16.768092 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.768061 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 07 10:11:16.768886 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.768871 2565 server.go:310] "Adding debug handlers to kubelet server" Sep 07 10:11:16.771288 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.771269 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 07 10:11:16.774212 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.774195 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 07 10:11:16.774905 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.774886 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 07 10:11:16.775040 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.775005 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:16.775620 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775603 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 07 10:11:16.775620 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775621 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 07 10:11:16.775728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775604 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 07 10:11:16.775728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775692 2565 reconstruct.go:97] "Volume reconstruction finished" Sep 07 10:11:16.775728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775700 2565 reconciler.go:29] "Reconciler: start to sync state" Sep 07 10:11:16.775833 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775799 2565 factory.go:153] Registering CRI-O factory Sep 07 10:11:16.775833 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775812 2565 factory.go:223] Registration of the crio container factory successfully Sep 07 10:11:16.775904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775857 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 07 10:11:16.775904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775864 2565 factory.go:55] Registering systemd factory Sep 07 10:11:16.775904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775869 2565 factory.go:223] Registration of the systemd container factory successfully Sep 07 10:11:16.775904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775882 2565 factory.go:103] Registering Raw factory Sep 07 10:11:16.775904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.775891 2565 manager.go:1196] Started watching for new ooms in manager Sep 07 10:11:16.776330 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.776314 2565 manager.go:349] Starting recovery of all containers Sep 07 10:11:16.780989 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.780898 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 07 10:11:16.780989 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.780925 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 07 10:11:16.782347 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.780984 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-168.ec2.internal.18d3022ebd725e32 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-168.ec2.internal,UID:ip-10-0-132-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-168.ec2.internal,},FirstTimestamp:2026-09-07 10:11:16.76635909 +0000 UTC m=+0.406867906,LastTimestamp:2026-09-07 10:11:16.76635909 +0000 UTC m=+0.406867906,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-168.ec2.internal,}" Sep 07 10:11:16.786696 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.786680 2565 manager.go:354] Recovery completed Sep 07 10:11:16.790038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.790013 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrhp5" Sep 07 10:11:16.792476 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.792465 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:11:16.794259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.794244 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:11:16.794313 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.794275 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:11:16.794313 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.794288 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:11:16.794795 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.794777 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 07 10:11:16.794795 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.794789 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 07 10:11:16.794795 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.794792 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:11:16.795016 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.794803 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 07 10:11:16.795504 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.795491 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:11:16.795564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.795518 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:11:16.795564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.795532 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:11:16.796048 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.795981 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-168.ec2.internal.18d3022ebf1c194a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-168.ec2.internal,UID:ip-10-0-132-168.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-168.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-168.ec2.internal,},FirstTimestamp:2026-09-07 10:11:16.794259786 +0000 UTC m=+0.434768602,LastTimestamp:2026-09-07 10:11:16.794259786 +0000 UTC m=+0.434768602,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-168.ec2.internal,}" Sep 07 10:11:16.796904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.796891 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrhp5" Sep 07 10:11:16.796971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.796904 2565 policy_none.go:49] "None policy: Start" Sep 07 10:11:16.796971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.796917 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 07 10:11:16.796971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.796944 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 07 10:11:16.798888 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.798876 2565 policy_none.go:47] "Start" Sep 07 10:11:16.833178 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.833164 2565 manager.go:337] "Starting Device Plugin manager" Sep 07 10:11:16.833264 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.833198 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 07 10:11:16.833264 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.833208 2565 server.go:87] "Starting device plugin registration server" Sep 07 10:11:16.833512 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.833496 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 07 10:11:16.833591 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.833512 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 07 10:11:16.833645 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.833605 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 07 10:11:16.833691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.833680 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 07 10:11:16.833691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.833689 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 07 10:11:16.834245 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.834227 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 07 10:11:16.834304 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.834273 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:16.881589 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.881560 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 07 10:11:16.883634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.882876 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 07 10:11:16.883634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.882926 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 07 10:11:16.883634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.882978 2565 kubelet.go:2448] "Starting kubelet main sync loop" Sep 07 10:11:16.883634 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.883020 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 07 10:11:16.885303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.885278 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:11:16.934271 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.934165 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:11:16.935144 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.935125 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:11:16.935226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.935169 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:11:16.935226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.935185 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:11:16.935226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.935208 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-168.ec2.internal" Sep 07 10:11:16.944156 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.944140 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-168.ec2.internal" Sep 07 10:11:16.944225 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.944165 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-168.ec2.internal\": node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:16.966313 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:16.966289 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:16.983750 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.983711 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"] Sep 07 10:11:16.983827 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.983798 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:11:16.985231 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.985216 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:11:16.985308 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.985243 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:11:16.985308 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.985253 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:11:16.987541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.987529 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:11:16.987673 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.987659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:16.987714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.987689 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:11:16.988401 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.988386 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:11:16.988456 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.988417 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:11:16.988456 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.988427 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:11:16.988528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.988389 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:11:16.988528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.988483 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:11:16.988528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.988496 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:11:16.990707 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.990687 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Sep 07 10:11:16.990794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.990710 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:11:16.991444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.991428 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:11:16.991532 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.991456 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:11:16.991532 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:16.991470 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:11:17.009551 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.009524 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-168.ec2.internal\" not found" node="ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.014180 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.014164 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-168.ec2.internal\" not found" node="ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.066599 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.066567 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.076876 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.076849 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea865860dc1ded4b85a7a2ca71f02386-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"ea865860dc1ded4b85a7a2ca71f02386\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.077000 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.076882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a83faa11e28477cb4f1e91aee3abab78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"a83faa11e28477cb4f1e91aee3abab78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.077000 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.076944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83faa11e28477cb4f1e91aee3abab78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"a83faa11e28477cb4f1e91aee3abab78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.167029 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.167000 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.177346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.177318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a83faa11e28477cb4f1e91aee3abab78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"a83faa11e28477cb4f1e91aee3abab78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.177415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.177388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83faa11e28477cb4f1e91aee3abab78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"a83faa11e28477cb4f1e91aee3abab78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.177415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.177393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83faa11e28477cb4f1e91aee3abab78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"a83faa11e28477cb4f1e91aee3abab78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.177415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.177404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a83faa11e28477cb4f1e91aee3abab78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"a83faa11e28477cb4f1e91aee3abab78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.177523 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.177421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea865860dc1ded4b85a7a2ca71f02386-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"ea865860dc1ded4b85a7a2ca71f02386\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.177523 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.177445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea865860dc1ded4b85a7a2ca71f02386-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"ea865860dc1ded4b85a7a2ca71f02386\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.267837 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.267760 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.312226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.312196 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.317206 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.317188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Sep 07 10:11:17.367908 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.367875 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.468350 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.468313 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.568926 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.568849 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.607951 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.607909 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:11:17.668994 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.668970 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.705386 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.705340 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 07 10:11:17.705840 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.705516 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 10:11:17.705840 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.705539 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 10:11:17.769581 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.769540 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.800011 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.799974 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 10:06:18 +0000 UTC" deadline="2028-05-21 07:12:19.5460505 +0000 UTC" Sep 07 10:11:17.800011 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:17.800001 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14925h1m1.746052612s" Sep 07 10:11:17.870294 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.870271 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:17.971103 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:17.971063 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:18.031101 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:18.031073 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea865860dc1ded4b85a7a2ca71f02386.slice/crio-6387139990eab869284ed4bf2abd34683c560eacf849f54a9cd2b05b1b4428c1 WatchSource:0}: Error finding container 6387139990eab869284ed4bf2abd34683c560eacf849f54a9cd2b05b1b4428c1: Status 404 returned error can't find the container with id 6387139990eab869284ed4bf2abd34683c560eacf849f54a9cd2b05b1b4428c1 Sep 07 10:11:18.031303 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:18.031280 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83faa11e28477cb4f1e91aee3abab78.slice/crio-6b94beda7b5c90b7dc84d5bb0700cd8c123e5062a4253af8085f8f0bbd825990 WatchSource:0}: Error finding container 6b94beda7b5c90b7dc84d5bb0700cd8c123e5062a4253af8085f8f0bbd825990: Status 404 returned error can't find the container with id 6b94beda7b5c90b7dc84d5bb0700cd8c123e5062a4253af8085f8f0bbd825990 Sep 07 10:11:18.035214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:18.035197 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:11:18.072140 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:18.072110 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found" Sep 07 10:11:18.125575 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:18.125503 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:11:18.175513 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:18.175170 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" Sep 07 10:11:18.187865 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:18.187843 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 10:11:18.188699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:18.188687 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" Sep 07 10:11:18.197316 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:18.197297 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 10:11:18.203726 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:18.203711 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:11:20.342122 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.342089 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:11:20.406414 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.406384 2565 apiserver.go:52] "Watching apiserver" Sep 07 10:11:20.413846 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.413818 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 07 10:11:20.414876 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.414851 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-nsmhj","kube-system/global-pull-secret-syncer-f4j6j","kube-system/konnectivity-agent-vtq76","kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2tpkb","openshift-image-registry/node-ca-nqhd9","openshift-multus/multus-n7z4k","openshift-network-operator/iptables-alerter-jztrk","openshift-ovn-kubernetes/ovnkube-node-8qhw7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29","openshift-dns/node-resolver-r9qwl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal","openshift-multus/multus-additional-cni-plugins-5c5vj","openshift-multus/network-metrics-daemon-lb6k7"] Sep 07 10:11:20.417380 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.417354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.419626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.419602 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.420135 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.420117 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 07 10:11:20.420310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.420296 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 07 10:11:20.420386 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.420333 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l7t4m\"" Sep 07 10:11:20.420439 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.420417 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 07 10:11:20.422274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.421750 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 07 10:11:20.422274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.421942 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:20.422274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.422024 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 07 10:11:20.422415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.422346 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rw82\"" Sep 07 10:11:20.424199 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.424178 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 07 10:11:20.424199 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.424198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:20.424315 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.424264 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:20.424497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.424482 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nhwjb\"" Sep 07 10:11:20.424632 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.424481 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 07 10:11:20.426465 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.426349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.428633 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.428615 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:11:20.428837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.428822 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 07 10:11:20.429000 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.428986 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48zjp\"" Sep 07 10:11:20.429255 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.429241 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 07 10:11:20.429359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.429340 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.429674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.429657 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 07 10:11:20.431566 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.431547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.434817 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.434780 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-csphj\"" Sep 07 10:11:20.434981 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.434942 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 07 10:11:20.435055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.434998 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.435108 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.435053 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nw5cn\"" Sep 07 10:11:20.435185 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.435113 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 07 10:11:20.435247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.435184 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 07 10:11:20.435247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.435240 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 07 10:11:20.435470 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.435444 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 07 10:11:20.435552 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.435534 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 07 10:11:20.436060 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.436032 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 07 10:11:20.436188 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.436117 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 07 10:11:20.436750 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.436716 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 07 10:11:20.436846 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.436822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.436944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.436862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-log-socket\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.436944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.436896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d7e04b2-a870-496f-8e4e-3003549f0113-ovn-node-metrics-cert\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.436944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.436926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-sys-fs\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.437134 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:20.437224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-systemd\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.437434 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-host-slash\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.437511 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4xp4\" (UniqueName: \"kubernetes.io/projected/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-kube-api-access-m4xp4\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.437572 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437525 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.437658 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-socket-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.437709 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-cni-bin\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.437761 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-ovnkube-config\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.437761 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437717 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 07 10:11:20.437761 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwx97\" (UniqueName: \"kubernetes.io/projected/6d7e04b2-a870-496f-8e4e-3003549f0113-kube-api-access-rwx97\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.437977 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-registration-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.437977 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/154eb144-57c7-41b2-8ce3-d7efce4a447e-hosts-file\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.437977 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154eb144-57c7-41b2-8ce3-d7efce4a447e-tmp-dir\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.438104 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.437981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-kubelet\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.438148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-node-log\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.438259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.438259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca012ec4-4ff9-41e3-9135-3f7b477068d4-agent-certs\") pod \"konnectivity-agent-vtq76\" (UID: \"ca012ec4-4ff9-41e3-9135-3f7b477068d4\") " pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:20.438259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba852866-ea68-4579-8b0a-8f2c843d8023-serviceca\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.438439 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-run-netns\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.438439 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-ovnkube-script-lib\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.438528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438439 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 07 10:11:20.438694 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438674 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 07 10:11:20.438858 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5nc\" (UniqueName: \"kubernetes.io/projected/ba852866-ea68-4579-8b0a-8f2c843d8023-kube-api-access-ml5nc\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.438931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-slash\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.438931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba852866-ea68-4579-8b0a-8f2c843d8023-host\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.439018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-ovn\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.439018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.438964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnm7\" (UniqueName: \"kubernetes.io/projected/05166e4c-db23-4e20-b194-d8adbf6ce954-kube-api-access-xbnm7\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:20.439018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-systemd-units\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.439018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439010 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-68tkj\"" Sep 07 10:11:20.439269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-var-lib-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.439269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-etc-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.439269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-run-ovn-kubernetes\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.439269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-cni-netd\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.439269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-env-overrides\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.439269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439200 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-iptables-alerter-script\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.439269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca012ec4-4ff9-41e3-9135-3f7b477068d4-konnectivity-ca\") pod \"konnectivity-agent-vtq76\" (UID: \"ca012ec4-4ff9-41e3-9135-3f7b477068d4\") " pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:20.439269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-device-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.439754 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439291 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.439754 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfnwx\" (UniqueName: \"kubernetes.io/projected/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-kube-api-access-sfnwx\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.439754 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qxll\" (UniqueName: \"kubernetes.io/projected/154eb144-57c7-41b2-8ce3-d7efce4a447e-kube-api-access-7qxll\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.439986 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.439963 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 07 10:11:20.440270 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.440255 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 07 10:11:20.440598 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.440581 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.440796 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.440772 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:20.442041 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.441849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:20.442041 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.441958 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:20.442284 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.442267 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 10:11:20.447323 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.447305 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.449677 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.449661 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8wpgm\"" Sep 07 10:11:20.449904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.449723 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 07 10:11:20.457068 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.457046 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.459787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.459290 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 07 10:11:20.459787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.459455 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6wlj" Sep 07 10:11:20.459787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.459563 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:11:20.459787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.459659 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hzrdz\"" Sep 07 10:11:20.471522 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.471480 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6wlj" Sep 07 10:11:20.531120 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.531096 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 07 10:11:20.539649 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-log-socket\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.539776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-sys-fs\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.539776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-etc-kubernetes\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.539776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539711 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysctl-conf\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.539776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539737 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-log-socket\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.539928 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-sys-fs\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.539928 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-host\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.539928 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-cnibin\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.539928 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-systemd\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.539928 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.540143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539951 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-socket-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.540143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-systemd\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.540143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540033 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.540143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.539974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-multus-certs\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.540143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-tuned\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.540143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.540143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-cni-bin\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/154eb144-57c7-41b2-8ce3-d7efce4a447e-hosts-file\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540200 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540212 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-cni-bin\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-node-log\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca012ec4-4ff9-41e3-9135-3f7b477068d4-agent-certs\") pod \"konnectivity-agent-vtq76\" (UID: \"ca012ec4-4ff9-41e3-9135-3f7b477068d4\") " pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-lib-modules\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-node-log\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540271 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/154eb144-57c7-41b2-8ce3-d7efce4a447e-hosts-file\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-ovnkube-script-lib\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-cni-multus\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-systemd\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540462 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-var-lib-kubelet\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540482 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-socket-dir-parent\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-hostroot\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-ovn\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnm7\" (UniqueName: \"kubernetes.io/projected/05166e4c-db23-4e20-b194-d8adbf6ce954-kube-api-access-xbnm7\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:20.540608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-run-ovn-kubernetes\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfnwx\" (UniqueName: \"kubernetes.io/projected/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-kube-api-access-sfnwx\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-cni-bin\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9278a09a-9c88-426b-b27c-c4808a5730c7-tmp\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540694 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0db793e3-82ec-40b8-ab3e-eddc545c4973-dbus\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-iptables-alerter-script\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca012ec4-4ff9-41e3-9135-3f7b477068d4-konnectivity-ca\") pod \"konnectivity-agent-vtq76\" (UID: \"ca012ec4-4ff9-41e3-9135-3f7b477068d4\") " pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysctl-d\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d7e04b2-a870-496f-8e4e-3003549f0113-ovn-node-metrics-cert\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-run\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540884 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.540969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-host-slash\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4xp4\" (UniqueName: \"kubernetes.io/projected/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-kube-api-access-m4xp4\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-conf-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ff68\" (UniqueName: \"kubernetes.io/projected/8eb29491-0135-47fe-9783-122b76372ef9-kube-api-access-7ff68\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.541226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6r5\" (UniqueName: \"kubernetes.io/projected/9278a09a-9c88-426b-b27c-c4808a5730c7-kube-api-access-jv6r5\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-ovnkube-config\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwx97\" (UniqueName: \"kubernetes.io/projected/6d7e04b2-a870-496f-8e4e-3003549f0113-kube-api-access-rwx97\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541230 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-registration-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154eb144-57c7-41b2-8ce3-d7efce4a447e-tmp-dir\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541288 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-system-cni-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-kubelet\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-kubelet\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba852866-ea68-4579-8b0a-8f2c843d8023-serviceca\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-cni-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysconfig\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-sys\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-run-netns\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5nc\" (UniqueName: \"kubernetes.io/projected/ba852866-ea68-4579-8b0a-8f2c843d8023-kube-api-access-ml5nc\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8eb29491-0135-47fe-9783-122b76372ef9-cni-binary-copy\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.541899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-k8s-cni-cncf-io\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-netns\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8eb29491-0135-47fe-9783-122b76372ef9-multus-daemon-config\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-system-cni-dir\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541696 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-slash\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba852866-ea68-4579-8b0a-8f2c843d8023-host\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-cni-binary-copy\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0db793e3-82ec-40b8-ab3e-eddc545c4973-kubelet-config\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-systemd-units\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-var-lib-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-etc-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-cni-netd\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-env-overrides\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qxll\" (UniqueName: \"kubernetes.io/projected/154eb144-57c7-41b2-8ce3-d7efce4a447e-kube-api-access-7qxll\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-modprobe-d\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-kubernetes\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.542646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-device-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.543346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.541993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.543346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.542016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-cnibin\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.543346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.542041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b2pt\" (UniqueName: \"kubernetes.io/projected/1f7afb78-4695-4bf1-9835-117132b6b761-kube-api-access-7b2pt\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.543346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.542663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-ovnkube-script-lib\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.543346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.542880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-ovn\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.543346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.543212 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-run-ovn-kubernetes\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.543935 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.543913 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-iptables-alerter-script\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.544393 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.544375 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca012ec4-4ff9-41e3-9135-3f7b477068d4-konnectivity-ca\") pod \"konnectivity-agent-vtq76\" (UID: \"ca012ec4-4ff9-41e3-9135-3f7b477068d4\") " pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:20.544491 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.544449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-run-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.544739 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.544722 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:20.544814 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.544796 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs podName:05166e4c-db23-4e20-b194-d8adbf6ce954 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:21.044769652 +0000 UTC m=+3.031049968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs") pod "network-metrics-daemon-lb6k7" (UID: "05166e4c-db23-4e20-b194-d8adbf6ce954") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:20.545007 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.544979 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-host-slash\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.545775 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.545738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-ovnkube-config\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.546031 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.546005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-registration-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.548441 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-kubelet\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.548496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.548884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba852866-ea68-4579-8b0a-8f2c843d8023-serviceca\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.548974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-run-netns\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-cni-netd\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549314 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-host-slash\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba852866-ea68-4579-8b0a-8f2c843d8023-host\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549409 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-systemd-units\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-var-lib-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549470 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d7e04b2-a870-496f-8e4e-3003549f0113-etc-openvswitch\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549644 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549765 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-device-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.550063 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.549843 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d7e04b2-a870-496f-8e4e-3003549f0113-env-overrides\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.550689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.550267 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 07 10:11:20.550689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.550620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-socket-dir\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.551013 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.550947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154eb144-57c7-41b2-8ce3-d7efce4a447e-tmp-dir\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.551551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.551462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"a83faa11e28477cb4f1e91aee3abab78","Type":"ContainerStarted","Data":"6b94beda7b5c90b7dc84d5bb0700cd8c123e5062a4253af8085f8f0bbd825990"} Sep 07 10:11:20.553854 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.553831 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" event={"ID":"ea865860dc1ded4b85a7a2ca71f02386","Type":"ContainerStarted","Data":"6387139990eab869284ed4bf2abd34683c560eacf849f54a9cd2b05b1b4428c1"} Sep 07 10:11:20.562967 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.560014 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4xp4\" (UniqueName: \"kubernetes.io/projected/582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372-kube-api-access-m4xp4\") pod \"iptables-alerter-jztrk\" (UID: \"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372\") " pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.562967 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.560337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfnwx\" (UniqueName: \"kubernetes.io/projected/df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb-kube-api-access-sfnwx\") pod \"aws-ebs-csi-driver-node-5ns29\" (UID: \"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.564413 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.564392 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d7e04b2-a870-496f-8e4e-3003549f0113-ovn-node-metrics-cert\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.564637 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.564611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca012ec4-4ff9-41e3-9135-3f7b477068d4-agent-certs\") pod \"konnectivity-agent-vtq76\" (UID: \"ca012ec4-4ff9-41e3-9135-3f7b477068d4\") " pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:20.564715 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.564693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnm7\" (UniqueName: \"kubernetes.io/projected/05166e4c-db23-4e20-b194-d8adbf6ce954-kube-api-access-xbnm7\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:20.565219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.565198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwx97\" (UniqueName: \"kubernetes.io/projected/6d7e04b2-a870-496f-8e4e-3003549f0113-kube-api-access-rwx97\") pod \"ovnkube-node-8qhw7\" (UID: \"6d7e04b2-a870-496f-8e4e-3003549f0113\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.565680 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.565320 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qxll\" (UniqueName: \"kubernetes.io/projected/154eb144-57c7-41b2-8ce3-d7efce4a447e-kube-api-access-7qxll\") pod \"node-resolver-r9qwl\" (UID: \"154eb144-57c7-41b2-8ce3-d7efce4a447e\") " pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.566733 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.566713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5nc\" (UniqueName: \"kubernetes.io/projected/ba852866-ea68-4579-8b0a-8f2c843d8023-kube-api-access-ml5nc\") pod \"node-ca-nqhd9\" (UID: \"ba852866-ea68-4579-8b0a-8f2c843d8023\") " pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.642480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8eb29491-0135-47fe-9783-122b76372ef9-cni-binary-copy\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.642480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-k8s-cni-cncf-io\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.642480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642444 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-netns\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.642480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8eb29491-0135-47fe-9783-122b76372ef9-multus-daemon-config\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.642480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-system-cni-dir\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642515 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-cni-binary-copy\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0db793e3-82ec-40b8-ab3e-eddc545c4973-kubelet-config\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642556 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-modprobe-d\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-kubernetes\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642593 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-cnibin\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2pt\" (UniqueName: \"kubernetes.io/projected/1f7afb78-4695-4bf1-9835-117132b6b761-kube-api-access-7b2pt\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-etc-kubernetes\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysctl-conf\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-host\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-cnibin\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-multus-certs\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-tuned\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.642807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-lib-modules\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-cni-multus\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642866 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-systemd\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-var-lib-kubelet\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-socket-dir-parent\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-hostroot\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-cni-bin\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.642985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9278a09a-9c88-426b-b27c-c4808a5730c7-tmp\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0db793e3-82ec-40b8-ab3e-eddc545c4973-dbus\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysctl-d\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-run\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-conf-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ff68\" (UniqueName: \"kubernetes.io/projected/8eb29491-0135-47fe-9783-122b76372ef9-kube-api-access-7ff68\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6r5\" (UniqueName: \"kubernetes.io/projected/9278a09a-9c88-426b-b27c-c4808a5730c7-kube-api-access-jv6r5\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-system-cni-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.643421 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-kubelet\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.644216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-cni-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.644216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysconfig\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.644216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-sys\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.644216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643470 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-sys\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.644216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.643975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8eb29491-0135-47fe-9783-122b76372ef9-cni-binary-copy\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.644216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.644036 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-k8s-cni-cncf-io\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.644216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.644069 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-netns\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.644520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.644410 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8eb29491-0135-47fe-9783-122b76372ef9-multus-daemon-config\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.644520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.644446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-system-cni-dir\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.644805 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-cni-binary-copy\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0db793e3-82ec-40b8-ab3e-eddc545c4973-kubelet-config\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-socket-dir-parent\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-var-lib-kubelet\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645297 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-hostroot\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-modprobe-d\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-kubernetes\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-cnibin\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0db793e3-82ec-40b8-ab3e-eddc545c4973-dbus\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645583 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysctl-d\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645632 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-run\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.645710 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.645763 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret podName:0db793e3-82ec-40b8-ab3e-eddc545c4973 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:21.145746168 +0000 UTC m=+3.132026494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret") pod "global-pull-secret-syncer-f4j6j" (UID: "0db793e3-82ec-40b8-ab3e-eddc545c4973") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645813 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-cni-multus\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645936 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-lib-modules\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.646647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-system-cni-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-kubelet\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysconfig\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646065 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-cni-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646245 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-multus-conf-dir\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646373 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-sysctl-conf\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646521 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-etc-kubernetes\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f7afb78-4695-4bf1-9835-117132b6b761-cnibin\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646583 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f7afb78-4695-4bf1-9835-117132b6b761-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.645413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-var-lib-cni-bin\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646590 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-host\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8eb29491-0135-47fe-9783-122b76372ef9-host-run-multus-certs\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.647473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.646653 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-systemd\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.648594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.648574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9278a09a-9c88-426b-b27c-c4808a5730c7-tmp\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.648680 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.648611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9278a09a-9c88-426b-b27c-c4808a5730c7-etc-tuned\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.654087 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.654068 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:11:20.654190 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.654095 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:11:20.654190 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.654108 2565 projected.go:196] Error preparing data for projected volume kube-api-access-nlznh for pod openshift-network-diagnostics/network-check-target-nsmhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:20.654190 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:20.654182 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh podName:94d5e68e-e01b-421b-b544-fe76fd212ed4 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:21.154145418 +0000 UTC m=+3.140425743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nlznh" (UniqueName: "kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh") pod "network-check-target-nsmhj" (UID: "94d5e68e-e01b-421b-b544-fe76fd212ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:20.657058 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.657038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ff68\" (UniqueName: \"kubernetes.io/projected/8eb29491-0135-47fe-9783-122b76372ef9-kube-api-access-7ff68\") pod \"multus-n7z4k\" (UID: \"8eb29491-0135-47fe-9783-122b76372ef9\") " pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.658470 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.658085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6r5\" (UniqueName: \"kubernetes.io/projected/9278a09a-9c88-426b-b27c-c4808a5730c7-kube-api-access-jv6r5\") pod \"tuned-2tpkb\" (UID: \"9278a09a-9c88-426b-b27c-c4808a5730c7\") " pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:20.658470 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.658100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2pt\" (UniqueName: \"kubernetes.io/projected/1f7afb78-4695-4bf1-9835-117132b6b761-kube-api-access-7b2pt\") pod \"multus-additional-cni-plugins-5c5vj\" (UID: \"1f7afb78-4695-4bf1-9835-117132b6b761\") " pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.728480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.728444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" Sep 07 10:11:20.737637 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.737613 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9qwl" Sep 07 10:11:20.749422 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.749404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:20.757665 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.757645 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jztrk" Sep 07 10:11:20.766216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.766194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqhd9" Sep 07 10:11:20.779030 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.778972 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:20.787046 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.787027 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" Sep 07 10:11:20.799806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.799785 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n7z4k" Sep 07 10:11:20.804548 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:20.804528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" Sep 07 10:11:21.046771 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:21.046689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:21.046893 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:21.046837 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:21.046927 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:21.046902 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs podName:05166e4c-db23-4e20-b194-d8adbf6ce954 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:22.046888131 +0000 UTC m=+4.033168442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs") pod "network-metrics-daemon-lb6k7" (UID: "05166e4c-db23-4e20-b194-d8adbf6ce954") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:21.147784 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:21.147740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:21.147954 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:21.147870 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:21.147954 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:21.147923 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret podName:0db793e3-82ec-40b8-ab3e-eddc545c4973 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:22.147909394 +0000 UTC m=+4.134189706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret") pod "global-pull-secret-syncer-f4j6j" (UID: "0db793e3-82ec-40b8-ab3e-eddc545c4973") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:21.248311 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:21.248278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:21.248469 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:21.248452 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:11:21.248530 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:21.248474 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:11:21.248530 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:21.248484 2565 projected.go:196] Error preparing data for projected volume kube-api-access-nlznh for pod openshift-network-diagnostics/network-check-target-nsmhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:21.248617 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:21.248538 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh podName:94d5e68e-e01b-421b-b544-fe76fd212ed4 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:22.24852078 +0000 UTC m=+4.234801095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlznh" (UniqueName: "kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh") pod "network-check-target-nsmhj" (UID: "94d5e68e-e01b-421b-b544-fe76fd212ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:21.474717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:21.474642 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:06:20 +0000 UTC" deadline="2028-03-01 20:16:27.361425525 +0000 UTC" Sep 07 10:11:21.474717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:21.474671 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12994h5m5.886757264s" Sep 07 10:11:21.719222 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.718999 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7e04b2_a870_496f_8e4e_3003549f0113.slice/crio-a3b08ae793a4dd8420cc812174adc7072670d140da1e5ac70e6a128189009e86 WatchSource:0}: Error finding container a3b08ae793a4dd8420cc812174adc7072670d140da1e5ac70e6a128189009e86: Status 404 returned error can't find the container with id a3b08ae793a4dd8420cc812174adc7072670d140da1e5ac70e6a128189009e86 Sep 07 10:11:21.721320 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.721140 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9278a09a_9c88_426b_b27c_c4808a5730c7.slice/crio-b241c7d4adf672bcbd07bff4abce397ca6c77026b21fa1e21c9d5039e6baffcd WatchSource:0}: Error finding container b241c7d4adf672bcbd07bff4abce397ca6c77026b21fa1e21c9d5039e6baffcd: Status 404 returned error can't find the container with id b241c7d4adf672bcbd07bff4abce397ca6c77026b21fa1e21c9d5039e6baffcd Sep 07 10:11:21.722063 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.721975 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7afb78_4695_4bf1_9835_117132b6b761.slice/crio-e91982fa4f3909f8bf97a47867f6b07c85771cbd2cfa1dc19f958204bd50e6ec WatchSource:0}: Error finding container e91982fa4f3909f8bf97a47867f6b07c85771cbd2cfa1dc19f958204bd50e6ec: Status 404 returned error can't find the container with id e91982fa4f3909f8bf97a47867f6b07c85771cbd2cfa1dc19f958204bd50e6ec Sep 07 10:11:21.725177 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.725143 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0f0e92_c8ea_4615_9ad7_1d8bfe93cbcb.slice/crio-94fe59d92e55ee5bece19fa2ea999b6d725234e00824a4ad507a78d9f205fd5f WatchSource:0}: Error finding container 94fe59d92e55ee5bece19fa2ea999b6d725234e00824a4ad507a78d9f205fd5f: Status 404 returned error can't find the container with id 94fe59d92e55ee5bece19fa2ea999b6d725234e00824a4ad507a78d9f205fd5f Sep 07 10:11:21.726083 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.725550 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba852866_ea68_4579_8b0a_8f2c843d8023.slice/crio-0969b48b580387c237f9c83c7c4b90e6d7bcdb4a02b0520604d2e33470fda9b9 WatchSource:0}: Error finding container 0969b48b580387c237f9c83c7c4b90e6d7bcdb4a02b0520604d2e33470fda9b9: Status 404 returned error can't find the container with id 0969b48b580387c237f9c83c7c4b90e6d7bcdb4a02b0520604d2e33470fda9b9 Sep 07 10:11:21.726764 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.726740 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb29491_0135_47fe_9783_122b76372ef9.slice/crio-e37a74e6f6048581ec6af68b3e54be1fb40e76b9d52b8c091f51f07c8eb6ddc5 WatchSource:0}: Error finding container e37a74e6f6048581ec6af68b3e54be1fb40e76b9d52b8c091f51f07c8eb6ddc5: Status 404 returned error can't find the container with id e37a74e6f6048581ec6af68b3e54be1fb40e76b9d52b8c091f51f07c8eb6ddc5 Sep 07 10:11:21.728603 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.728521 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca012ec4_4ff9_41e3_9135_3f7b477068d4.slice/crio-7d6c9e1468115c37f1641e1d1a655c62c1b28a12b3e8586d6356251bc489273d WatchSource:0}: Error finding container 7d6c9e1468115c37f1641e1d1a655c62c1b28a12b3e8586d6356251bc489273d: Status 404 returned error can't find the container with id 7d6c9e1468115c37f1641e1d1a655c62c1b28a12b3e8586d6356251bc489273d Sep 07 10:11:21.730323 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.730299 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582548f4_9c4f_4e9b_b1a2_ed5ca9dbf372.slice/crio-5951b32c6b0fd68fd38f5d7e0a3ce7decaa16f16c3ef2ec417b97f93ea301622 WatchSource:0}: Error finding container 5951b32c6b0fd68fd38f5d7e0a3ce7decaa16f16c3ef2ec417b97f93ea301622: Status 404 returned error can't find the container with id 5951b32c6b0fd68fd38f5d7e0a3ce7decaa16f16c3ef2ec417b97f93ea301622 Sep 07 10:11:21.731630 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:21.731197 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154eb144_57c7_41b2_8ce3_d7efce4a447e.slice/crio-999bca2c8169663377ef3a62d6bc5bad35d762039d397126f1f25d623dee9fdf WatchSource:0}: Error finding container 999bca2c8169663377ef3a62d6bc5bad35d762039d397126f1f25d623dee9fdf: Status 404 returned error can't find the container with id 999bca2c8169663377ef3a62d6bc5bad35d762039d397126f1f25d623dee9fdf Sep 07 10:11:22.054430 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.054405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:22.054617 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.054597 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:22.054744 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.054725 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs podName:05166e4c-db23-4e20-b194-d8adbf6ce954 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:24.054708741 +0000 UTC m=+6.040989057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs") pod "network-metrics-daemon-lb6k7" (UID: "05166e4c-db23-4e20-b194-d8adbf6ce954") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:22.155751 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.155708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:22.155924 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.155874 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:22.156012 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.155997 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret podName:0db793e3-82ec-40b8-ab3e-eddc545c4973 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:24.155974261 +0000 UTC m=+6.142254597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret") pod "global-pull-secret-syncer-f4j6j" (UID: "0db793e3-82ec-40b8-ab3e-eddc545c4973") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:22.186255 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.186231 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:11:22.257038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.256962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:22.257216 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.257118 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:11:22.257294 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.257225 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:11:22.257294 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.257242 2565 projected.go:196] Error preparing data for projected volume kube-api-access-nlznh for pod openshift-network-diagnostics/network-check-target-nsmhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:22.257393 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.257306 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh podName:94d5e68e-e01b-421b-b544-fe76fd212ed4 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:24.257287646 +0000 UTC m=+6.243567973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlznh" (UniqueName: "kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh") pod "network-check-target-nsmhj" (UID: "94d5e68e-e01b-421b-b544-fe76fd212ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:22.475634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.475568 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:06:20 +0000 UTC" deadline="2028-05-23 18:46:06.985442542 +0000 UTC" Sep 07 10:11:22.475634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.475632 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14984h34m44.509814593s" Sep 07 10:11:22.540411 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.540329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:22.540571 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.540463 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:22.540866 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.540841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:22.540951 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.540935 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:22.541010 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.541000 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:22.541093 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:22.541075 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:22.574605 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.574516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n7z4k" event={"ID":"8eb29491-0135-47fe-9783-122b76372ef9","Type":"ContainerStarted","Data":"e37a74e6f6048581ec6af68b3e54be1fb40e76b9d52b8c091f51f07c8eb6ddc5"} Sep 07 10:11:22.583790 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.583722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" event={"ID":"1f7afb78-4695-4bf1-9835-117132b6b761","Type":"ContainerStarted","Data":"e91982fa4f3909f8bf97a47867f6b07c85771cbd2cfa1dc19f958204bd50e6ec"} Sep 07 10:11:22.602291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.602254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" event={"ID":"9278a09a-9c88-426b-b27c-c4808a5730c7","Type":"ContainerStarted","Data":"b241c7d4adf672bcbd07bff4abce397ca6c77026b21fa1e21c9d5039e6baffcd"} Sep 07 10:11:22.617942 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.617902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" event={"ID":"ea865860dc1ded4b85a7a2ca71f02386","Type":"ContainerStarted","Data":"0855c04a4b2d0f289232ae13ebf613c5885ebf2aa3bacd27dbf5d2c620f9951b"} Sep 07 10:11:22.632177 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.632135 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9qwl" event={"ID":"154eb144-57c7-41b2-8ce3-d7efce4a447e","Type":"ContainerStarted","Data":"999bca2c8169663377ef3a62d6bc5bad35d762039d397126f1f25d623dee9fdf"} Sep 07 10:11:22.636088 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.636037 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" podStartSLOduration=3.636020608 podStartE2EDuration="3.636020608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:22.635803343 +0000 UTC m=+4.622083677" watchObservedRunningTime="2026-09-07 10:11:22.636020608 +0000 UTC m=+4.622300945" Sep 07 10:11:22.640789 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.640220 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jztrk" event={"ID":"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372","Type":"ContainerStarted","Data":"5951b32c6b0fd68fd38f5d7e0a3ce7decaa16f16c3ef2ec417b97f93ea301622"} Sep 07 10:11:22.645433 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.645408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtq76" event={"ID":"ca012ec4-4ff9-41e3-9135-3f7b477068d4","Type":"ContainerStarted","Data":"7d6c9e1468115c37f1641e1d1a655c62c1b28a12b3e8586d6356251bc489273d"} Sep 07 10:11:22.650115 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.650091 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqhd9" event={"ID":"ba852866-ea68-4579-8b0a-8f2c843d8023","Type":"ContainerStarted","Data":"0969b48b580387c237f9c83c7c4b90e6d7bcdb4a02b0520604d2e33470fda9b9"} Sep 07 10:11:22.654675 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.654641 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" event={"ID":"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb","Type":"ContainerStarted","Data":"94fe59d92e55ee5bece19fa2ea999b6d725234e00824a4ad507a78d9f205fd5f"} Sep 07 10:11:22.661507 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.661482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"a3b08ae793a4dd8420cc812174adc7072670d140da1e5ac70e6a128189009e86"} Sep 07 10:11:22.683141 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.683112 2565 generic.go:356] "Generic (PLEG): container finished" podID="a83faa11e28477cb4f1e91aee3abab78" containerID="20897125c262d0e333a56746791e0e38ac861868cd5af847db3ba1dd0b9d2b0a" exitCode=0 Sep 07 10:11:22.683267 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:22.683154 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"a83faa11e28477cb4f1e91aee3abab78","Type":"ContainerDied","Data":"20897125c262d0e333a56746791e0e38ac861868cd5af847db3ba1dd0b9d2b0a"} Sep 07 10:11:23.719511 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:23.719293 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"a83faa11e28477cb4f1e91aee3abab78","Type":"ContainerStarted","Data":"f59cb6ec5c7f42e0fddaea180a81c166f2fce4d4e1ae787f60af24e976c9c94a"} Sep 07 10:11:23.734782 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:23.734661 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" podStartSLOduration=4.734636912 podStartE2EDuration="4.734636912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:23.733852994 +0000 UTC m=+5.720133328" watchObservedRunningTime="2026-09-07 10:11:23.734636912 +0000 UTC m=+5.720917247" Sep 07 10:11:24.071763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:24.071236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:24.071763 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.071382 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:24.071763 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.071441 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs podName:05166e4c-db23-4e20-b194-d8adbf6ce954 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:28.071423481 +0000 UTC m=+10.057703807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs") pod "network-metrics-daemon-lb6k7" (UID: "05166e4c-db23-4e20-b194-d8adbf6ce954") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:24.172072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:24.172035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:24.172254 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.172208 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:24.172302 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.172273 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret podName:0db793e3-82ec-40b8-ab3e-eddc545c4973 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:28.172255393 +0000 UTC m=+10.158535719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret") pod "global-pull-secret-syncer-f4j6j" (UID: "0db793e3-82ec-40b8-ab3e-eddc545c4973") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:24.273005 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:24.272962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:24.273211 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.273125 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:11:24.273211 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.273148 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:11:24.273211 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.273174 2565 projected.go:196] Error preparing data for projected volume kube-api-access-nlznh for pod openshift-network-diagnostics/network-check-target-nsmhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:24.273326 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.273233 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh podName:94d5e68e-e01b-421b-b544-fe76fd212ed4 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:28.273214118 +0000 UTC m=+10.259494444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlznh" (UniqueName: "kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh") pod "network-check-target-nsmhj" (UID: "94d5e68e-e01b-421b-b544-fe76fd212ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:24.547177 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:24.541671 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:24.547177 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.541844 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:24.547177 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:24.542303 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:24.547177 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.542418 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:24.547177 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:24.542491 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:24.547177 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:24.542619 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:26.538312 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:26.538279 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:26.538660 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:26.538410 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:26.538721 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:26.538704 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:26.538785 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:26.538769 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:26.538886 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:26.538866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:26.539007 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:26.538986 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:28.109230 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:28.109196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:28.109710 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.109356 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:28.109710 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.109423 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs podName:05166e4c-db23-4e20-b194-d8adbf6ce954 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:36.109402002 +0000 UTC m=+18.095682321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs") pod "network-metrics-daemon-lb6k7" (UID: "05166e4c-db23-4e20-b194-d8adbf6ce954") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:28.210359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:28.210321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:28.210526 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.210444 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:28.210526 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.210519 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret podName:0db793e3-82ec-40b8-ab3e-eddc545c4973 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:36.210496769 +0000 UTC m=+18.196777084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret") pod "global-pull-secret-syncer-f4j6j" (UID: "0db793e3-82ec-40b8-ab3e-eddc545c4973") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:28.311330 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:28.311301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:28.311488 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.311457 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:11:28.311488 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.311474 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:11:28.311488 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.311483 2565 projected.go:196] Error preparing data for projected volume kube-api-access-nlznh for pod openshift-network-diagnostics/network-check-target-nsmhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:28.311608 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.311536 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh podName:94d5e68e-e01b-421b-b544-fe76fd212ed4 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:36.31152102 +0000 UTC m=+18.297801333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlznh" (UniqueName: "kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh") pod "network-check-target-nsmhj" (UID: "94d5e68e-e01b-421b-b544-fe76fd212ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:28.538783 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:28.538750 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:28.538965 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.538870 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:28.539372 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:28.539346 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:28.539516 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.539438 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:28.539516 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:28.539508 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:28.539621 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:28.539604 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:30.539675 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:30.539645 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:30.540131 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:30.539755 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:30.540131 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:30.540048 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:30.540131 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:30.540107 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:30.540348 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:30.540193 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:30.540348 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:30.540276 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:32.537914 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:32.537678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:32.538368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:32.537678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:32.538368 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:32.538007 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:32.538368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:32.537716 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:32.538368 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:32.538093 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:32.538368 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:32.538143 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:34.537603 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:34.537568 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:34.538099 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:34.537569 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:34.538099 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:34.537681 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:34.538099 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:34.537705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:34.538099 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:34.537765 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:34.538099 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:34.537828 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:36.168529 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:36.168489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:36.169086 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.168663 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:36.169086 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.168741 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs podName:05166e4c-db23-4e20-b194-d8adbf6ce954 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:52.168718191 +0000 UTC m=+34.154998519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs") pod "network-metrics-daemon-lb6k7" (UID: "05166e4c-db23-4e20-b194-d8adbf6ce954") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:36.269446 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:36.269412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:36.269635 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.269581 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:36.269684 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.269654 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret podName:0db793e3-82ec-40b8-ab3e-eddc545c4973 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:52.26963358 +0000 UTC m=+34.255913897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret") pod "global-pull-secret-syncer-f4j6j" (UID: "0db793e3-82ec-40b8-ab3e-eddc545c4973") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:36.370755 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:36.370712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:36.370943 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.370859 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:11:36.370943 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.370877 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:11:36.370943 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.370889 2565 projected.go:196] Error preparing data for projected volume kube-api-access-nlznh for pod openshift-network-diagnostics/network-check-target-nsmhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:36.370943 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.370943 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh podName:94d5e68e-e01b-421b-b544-fe76fd212ed4 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:52.370925473 +0000 UTC m=+34.357205799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlznh" (UniqueName: "kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh") pod "network-check-target-nsmhj" (UID: "94d5e68e-e01b-421b-b544-fe76fd212ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:36.538285 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:36.538255 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:36.538481 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:36.538254 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:36.538481 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.538381 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:36.538481 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.538469 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:36.538638 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:36.538264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:36.538638 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:36.538555 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:38.538910 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:38.538882 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:38.539409 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:38.538964 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:38.539409 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:38.538985 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:38.539409 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:38.539126 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:38.539409 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:38.539182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:38.539409 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:38.539245 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:39.753604 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.753360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9qwl" event={"ID":"154eb144-57c7-41b2-8ce3-d7efce4a447e","Type":"ContainerStarted","Data":"8f82d244f029189935236960bbf25f8bee2a3e6a1e4d32f18931d2f85367bd97"} Sep 07 10:11:39.755116 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.755086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtq76" event={"ID":"ca012ec4-4ff9-41e3-9135-3f7b477068d4","Type":"ContainerStarted","Data":"e1d24cfb5c4a43e43b8b8f54ea17280d7bb17834f126ab16d1dfa1b69c80c587"} Sep 07 10:11:39.757091 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.756903 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqhd9" event={"ID":"ba852866-ea68-4579-8b0a-8f2c843d8023","Type":"ContainerStarted","Data":"ff9f910b1ef86adae76e401e70fe58bd3472d1f695f9966e2acb5bc21d581b60"} Sep 07 10:11:39.758564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.758543 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" event={"ID":"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb","Type":"ContainerStarted","Data":"8872b543c00d0a5a89e11198304a3cbb504fecbe9fb6e9227cc1bcf99d57f6ee"} Sep 07 10:11:39.760992 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.760969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"b2374c9f629e88ab322bf128ff62aa23473fc79f70bbe77c4e7ea5063ea6af62"} Sep 07 10:11:39.761109 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.761038 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"97e69e95dd5044ff1cf90360bc6ad233c9007747dc26d161e66c0595fd6574e8"} Sep 07 10:11:39.765683 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.765630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n7z4k" event={"ID":"8eb29491-0135-47fe-9783-122b76372ef9","Type":"ContainerStarted","Data":"c4146b782418a8641294219d292c8789aef4db900690f3bb44eacba9452ab24e"} Sep 07 10:11:39.767639 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.767611 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f7afb78-4695-4bf1-9835-117132b6b761" containerID="37d17b2844d427ae3fb80b191ae998fde3c02f90e989dfbc1d4acdd579d358cb" exitCode=0 Sep 07 10:11:39.767752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.767710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" event={"ID":"1f7afb78-4695-4bf1-9835-117132b6b761","Type":"ContainerDied","Data":"37d17b2844d427ae3fb80b191ae998fde3c02f90e989dfbc1d4acdd579d358cb"} Sep 07 10:11:39.768472 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.768075 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r9qwl" podStartSLOduration=4.11105378 podStartE2EDuration="21.768060192s" totalImagesPullingTime="17.657006412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:39.76798551 +0000 UTC m=+21.754265845" watchObservedRunningTime="2026-09-07 10:11:39.768060192 +0000 UTC m=+21.754340527" Sep 07 10:11:39.770305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.770277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" event={"ID":"9278a09a-9c88-426b-b27c-c4808a5730c7","Type":"ContainerStarted","Data":"cbb587f371640097c43d3c5a6366968fdadd441c6763caa5595c54c4f2513c34"} Sep 07 10:11:39.800822 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.800774 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n7z4k" podStartSLOduration=4.059495892 podStartE2EDuration="21.80075808s" totalImagesPullingTime="17.741262188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:39.800734744 +0000 UTC m=+21.787015089" watchObservedRunningTime="2026-09-07 10:11:39.80075808 +0000 UTC m=+21.787038413" Sep 07 10:11:39.814446 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.814405 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vtq76" podStartSLOduration=4.157385903 podStartE2EDuration="21.814391545s" totalImagesPullingTime="17.657005642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:39.814286494 +0000 UTC m=+21.800566825" watchObservedRunningTime="2026-09-07 10:11:39.814391545 +0000 UTC m=+21.800671879" Sep 07 10:11:39.828437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.828394 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nqhd9" podStartSLOduration=4.207520825 podStartE2EDuration="21.828378167s" totalImagesPullingTime="17.620857342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:39.827877501 +0000 UTC m=+21.814157835" watchObservedRunningTime="2026-09-07 10:11:39.828378167 +0000 UTC m=+21.814658502" Sep 07 10:11:39.843524 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:39.843477 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2tpkb" podStartSLOduration=3.151667653 podStartE2EDuration="20.843457113s" totalImagesPullingTime="17.69178946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:39.843324815 +0000 UTC m=+21.829605149" watchObservedRunningTime="2026-09-07 10:11:39.843457113 +0000 UTC m=+21.829737442" Sep 07 10:11:40.538279 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:40.538245 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:40.538496 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:40.538246 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:40.538496 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:40.538342 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:40.538496 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:40.538357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:40.538496 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:40.538434 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:40.538716 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:40.538501 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:40.776238 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:40.776189 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"5bbe625df5f4b672c9288387a6c14b553567dbda840e13b8b23775c592a50e08"} Sep 07 10:11:40.776941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:40.776303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"210a8f7d843c8fec0a5cff7c14e56f81e76c84a7872125d589c96f217cc79887"} Sep 07 10:11:40.776941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:40.776341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"2ee2aa7ecc93c7e1f91ac7decc3821829692e4a9bf7e54fc3cb214423d1f1eb4"} Sep 07 10:11:40.776941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:40.776370 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"d66fbfb7f2fccf899602eb189ca515d3a5b0e8d92d77f4cc289f3186b4802d40"} Sep 07 10:11:41.345170 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:41.345129 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 07 10:11:41.506401 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:41.506289 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T10:11:41.3451513Z","UUID":"c959c937-00bd-4f5d-bcbc-493f32f49b9e","Handler":null,"Name":"","Endpoint":""} Sep 07 10:11:41.507966 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:41.507924 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 07 10:11:41.507966 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:41.507959 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 07 10:11:41.779504 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:41.779253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jztrk" event={"ID":"582548f4-9c4f-4e9b-b1a2-ed5ca9dbf372","Type":"ContainerStarted","Data":"4d0822309b7e7344e02230ff163b681b59c30ffada1420075b901f20f3e7a4eb"} Sep 07 10:11:41.781014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:41.780991 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" event={"ID":"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb","Type":"ContainerStarted","Data":"ffd0819147bed483042c1457e4fff07c0ebe7f76adba5e961ea02a9ee2286e41"} Sep 07 10:11:41.793346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:41.793300 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jztrk" podStartSLOduration=6.148000861 podStartE2EDuration="23.793287236s" totalImagesPullingTime="17.645286375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:41.793221558 +0000 UTC m=+23.779501893" watchObservedRunningTime="2026-09-07 10:11:41.793287236 +0000 UTC m=+23.779567570" Sep 07 10:11:42.538022 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:42.537987 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:42.538236 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:42.537987 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:42.538236 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:42.538122 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:42.538236 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:42.538120 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:42.538370 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:42.538276 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:42.538370 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:42.538362 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:42.786542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:42.786510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" event={"ID":"df0f0e92-c8ea-4615-9ad7-1d8bfe93cbcb","Type":"ContainerStarted","Data":"2609e2425dc16a9fa9a73d18264a168a49d238830ffaeaec7d87bf8ac8badce4"} Sep 07 10:11:42.790585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:42.790520 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"88b944f19dc09a8356811837d01ac1051b1a61b1b13249fd40b4dba34a264122"} Sep 07 10:11:43.217339 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:43.217258 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:43.218012 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:43.217987 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:43.792942 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:43.792911 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:43.793920 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:43.793898 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:11:43.810954 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:43.810908 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ns29" podStartSLOduration=5.249327444 podStartE2EDuration="25.810895317s" totalImagesPullingTime="20.561567873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:43.810357298 +0000 UTC m=+25.796637646" watchObservedRunningTime="2026-09-07 10:11:43.810895317 +0000 UTC m=+25.797175651" Sep 07 10:11:44.540754 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:44.540714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:44.540754 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:44.540745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:44.540990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:44.540745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:44.540990 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:44.540834 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:44.540990 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:44.540924 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:44.541126 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:44.540990 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:45.800388 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:45.800177 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" event={"ID":"6d7e04b2-a870-496f-8e4e-3003549f0113","Type":"ContainerStarted","Data":"024eb014cf439bf60695bb9c1903a32520862e8898d0754d962690d19a4e9c94"} Sep 07 10:11:45.800933 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:45.800447 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:45.802132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:45.802106 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f7afb78-4695-4bf1-9835-117132b6b761" containerID="75c3f466e6397b43282e380390e671a90305195a644165ea3dc6da8d95c30ce2" exitCode=0 Sep 07 10:11:45.802265 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:45.802196 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" event={"ID":"1f7afb78-4695-4bf1-9835-117132b6b761","Type":"ContainerDied","Data":"75c3f466e6397b43282e380390e671a90305195a644165ea3dc6da8d95c30ce2"} Sep 07 10:11:45.816623 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:45.816600 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:45.829089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:45.829054 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" podStartSLOduration=10.10877087 podStartE2EDuration="27.829041321s" totalImagesPullingTime="17.720270451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:45.827822845 +0000 UTC m=+27.814103179" watchObservedRunningTime="2026-09-07 10:11:45.829041321 +0000 UTC m=+27.815321632" Sep 07 10:11:45.973770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:45.973728 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:46.541008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.540981 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:46.541188 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.540981 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:46.541188 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:46.541099 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:46.541310 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:46.541225 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:46.541310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.540980 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:46.541397 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:46.541326 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:46.805187 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.805081 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:46.821747 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.821716 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:11:46.950332 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.950308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nsmhj"] Sep 07 10:11:46.950463 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.950401 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:46.950512 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:46.950481 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:46.955320 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.955210 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f4j6j"] Sep 07 10:11:46.955559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.955517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:46.955879 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:46.955848 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:46.956393 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.955992 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lb6k7"] Sep 07 10:11:46.956393 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:46.956113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:46.956393 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:46.956248 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:47.809304 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:47.809274 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f7afb78-4695-4bf1-9835-117132b6b761" containerID="31515065485f808bcb8e4597907b1698846541718444219eccabc7033b75f1a4" exitCode=0 Sep 07 10:11:47.809696 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:47.809367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" event={"ID":"1f7afb78-4695-4bf1-9835-117132b6b761","Type":"ContainerDied","Data":"31515065485f808bcb8e4597907b1698846541718444219eccabc7033b75f1a4"} Sep 07 10:11:48.538920 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:48.538881 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:48.539075 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:48.538972 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:48.539075 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:48.538985 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:48.539075 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:48.539002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:48.539281 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:48.539106 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:48.539281 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:48.539210 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:49.816856 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:49.816683 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f7afb78-4695-4bf1-9835-117132b6b761" containerID="b00e367d31b3daf67963c1c1b0eac93385720486734f26765b7e3dc5a54a9176" exitCode=0 Sep 07 10:11:49.816856 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:49.816767 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" event={"ID":"1f7afb78-4695-4bf1-9835-117132b6b761","Type":"ContainerDied","Data":"b00e367d31b3daf67963c1c1b0eac93385720486734f26765b7e3dc5a54a9176"} Sep 07 10:11:50.541204 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:50.541172 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:50.541385 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:50.541218 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:50.541385 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:50.541176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:50.541385 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:50.541307 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4j6j" podUID="0db793e3-82ec-40b8-ab3e-eddc545c4973" Sep 07 10:11:50.541560 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:50.541425 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lb6k7" podUID="05166e4c-db23-4e20-b194-d8adbf6ce954" Sep 07 10:11:50.541560 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:50.541505 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsmhj" podUID="94d5e68e-e01b-421b-b544-fe76fd212ed4" Sep 07 10:11:52.186949 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.186874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:52.187378 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.187017 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:52.187378 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.187082 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs podName:05166e4c-db23-4e20-b194-d8adbf6ce954 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:24.187062005 +0000 UTC m=+66.173342334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs") pod "network-metrics-daemon-lb6k7" (UID: "05166e4c-db23-4e20-b194-d8adbf6ce954") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:11:52.287422 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.287385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:52.287604 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.287523 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:52.287604 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.287582 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret podName:0db793e3-82ec-40b8-ab3e-eddc545c4973 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:24.287569277 +0000 UTC m=+66.273849589 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret") pod "global-pull-secret-syncer-f4j6j" (UID: "0db793e3-82ec-40b8-ab3e-eddc545c4973") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:11:52.314547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.314510 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeReady" Sep 07 10:11:52.314686 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.314661 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 07 10:11:52.348307 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.348275 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78f9bbc69b-rrt8w"] Sep 07 10:11:52.351821 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.351794 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.354382 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.354231 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 07 10:11:52.354568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.354541 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 07 10:11:52.354568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.354563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hpk4\"" Sep 07 10:11:52.355654 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.355636 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 07 10:11:52.360357 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.360113 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 07 10:11:52.362833 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.362813 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bfpfk"] Sep 07 10:11:52.366759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.366738 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f9bbc69b-rrt8w"] Sep 07 10:11:52.366854 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.366771 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pbd8k"] Sep 07 10:11:52.366896 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.366872 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.369359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.369344 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8d2\"" Sep 07 10:11:52.369461 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.369344 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 07 10:11:52.369461 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.369344 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 07 10:11:52.370680 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.370660 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pbd8k" Sep 07 10:11:52.372792 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.372753 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 07 10:11:52.372915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.372866 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 07 10:11:52.375997 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.375957 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 07 10:11:52.376099 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.376039 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gq9jw\"" Sep 07 10:11:52.377485 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.377444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bfpfk"] Sep 07 10:11:52.378378 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.378350 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pbd8k"] Sep 07 10:11:52.388301 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.387758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:52.388301 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.387929 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:11:52.388301 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.387950 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:11:52.388301 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.387962 2565 projected.go:196] Error preparing data for projected volume kube-api-access-nlznh for pod openshift-network-diagnostics/network-check-target-nsmhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:52.388301 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.388006 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh podName:94d5e68e-e01b-421b-b544-fe76fd212ed4 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:24.387993913 +0000 UTC m=+66.374274224 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlznh" (UniqueName: "kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh") pod "network-check-target-nsmhj" (UID: "94d5e68e-e01b-421b-b544-fe76fd212ed4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:11:52.488431 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.488431 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-installation-pull-secrets\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.488634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjp8\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-kube-api-access-kkjp8\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.488634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.488634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4496a2f9-ed1d-4e70-919d-b4968349702f-ca-trust-extracted\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.488634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a78daa9-fda7-4dbc-9465-17e251db07f6-tmp-dir\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.488634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-bound-sa-token\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.488634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488625 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f1d9389-f077-4289-92f0-19197641f90c-cert\") pod \"ingress-canary-pbd8k\" (UID: \"9f1d9389-f077-4289-92f0-19197641f90c\") " pod="openshift-ingress-canary/ingress-canary-pbd8k" Sep 07 10:11:52.488803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488651 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp96\" (UniqueName: \"kubernetes.io/projected/7a78daa9-fda7-4dbc-9465-17e251db07f6-kube-api-access-tzp96\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.488803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-image-registry-private-configuration\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.488803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488725 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-certificates\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.488803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-trusted-ca\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.488803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqjh\" (UniqueName: \"kubernetes.io/projected/9f1d9389-f077-4289-92f0-19197641f90c-kube-api-access-frqjh\") pod \"ingress-canary-pbd8k\" (UID: \"9f1d9389-f077-4289-92f0-19197641f90c\") " pod="openshift-ingress-canary/ingress-canary-pbd8k" Sep 07 10:11:52.488803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.488797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a78daa9-fda7-4dbc-9465-17e251db07f6-config-volume\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.500867 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.500835 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vtq76_ca012ec4-4ff9-41e3-9135-3f7b477068d4/konnectivity-agent/0.log" limit=1024 Sep 07 10:11:52.538226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.538201 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:11:52.538226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.538223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:11:52.538437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.538405 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:11:52.541490 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.541459 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 10:11:52.541490 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.541468 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cp5xp\"" Sep 07 10:11:52.541690 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.541494 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 10:11:52.541690 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.541461 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 10:11:52.541690 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.541497 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9xn8\"" Sep 07 10:11:52.541848 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.541811 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 10:11:52.588422 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.588391 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vtq76_ca012ec4-4ff9-41e3-9135-3f7b477068d4/konnectivity-agent/0.log" limit=1024 Sep 07 10:11:52.589246 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-image-registry-private-configuration\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.589345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-certificates\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.589345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589294 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-trusted-ca\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.589458 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqjh\" (UniqueName: \"kubernetes.io/projected/9f1d9389-f077-4289-92f0-19197641f90c-kube-api-access-frqjh\") pod \"ingress-canary-pbd8k\" (UID: \"9f1d9389-f077-4289-92f0-19197641f90c\") " pod="openshift-ingress-canary/ingress-canary-pbd8k" Sep 07 10:11:52.589458 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a78daa9-fda7-4dbc-9465-17e251db07f6-config-volume\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.589568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.589568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-installation-pull-secrets\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.589568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjp8\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-kube-api-access-kkjp8\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.589568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.589748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4496a2f9-ed1d-4e70-919d-b4968349702f-ca-trust-extracted\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.589748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a78daa9-fda7-4dbc-9465-17e251db07f6-tmp-dir\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.589748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-bound-sa-token\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.589748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f1d9389-f077-4289-92f0-19197641f90c-cert\") pod \"ingress-canary-pbd8k\" (UID: \"9f1d9389-f077-4289-92f0-19197641f90c\") " pod="openshift-ingress-canary/ingress-canary-pbd8k" Sep 07 10:11:52.589748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.589694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp96\" (UniqueName: \"kubernetes.io/projected/7a78daa9-fda7-4dbc-9465-17e251db07f6-kube-api-access-tzp96\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.590118 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.590080 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 10:11:52.590118 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.590109 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9bbc69b-rrt8w: secret "image-registry-tls" not found Sep 07 10:11:52.590264 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.590154 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 10:11:52.590264 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.590233 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a78daa9-fda7-4dbc-9465-17e251db07f6-tmp-dir\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.590340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.590311 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a78daa9-fda7-4dbc-9465-17e251db07f6-config-volume\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.590383 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.590375 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls podName:4496a2f9-ed1d-4e70-919d-b4968349702f nodeName:}" failed. No retries permitted until 2026-09-07 10:11:53.090356082 +0000 UTC m=+35.076636407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls") pod "image-registry-78f9bbc69b-rrt8w" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f") : secret "image-registry-tls" not found Sep 07 10:11:52.590433 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:52.590392 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls podName:7a78daa9-fda7-4dbc-9465-17e251db07f6 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:53.090384207 +0000 UTC m=+35.076664519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls") pod "dns-default-bfpfk" (UID: "7a78daa9-fda7-4dbc-9465-17e251db07f6") : secret "dns-default-metrics-tls" not found Sep 07 10:11:52.590485 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.590473 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4496a2f9-ed1d-4e70-919d-b4968349702f-ca-trust-extracted\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.590826 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.590788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-certificates\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.591129 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.591106 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-trusted-ca\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.594011 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.593988 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-image-registry-private-configuration\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.594011 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.594005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-installation-pull-secrets\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.594130 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.593998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f1d9389-f077-4289-92f0-19197641f90c-cert\") pod \"ingress-canary-pbd8k\" (UID: \"9f1d9389-f077-4289-92f0-19197641f90c\") " pod="openshift-ingress-canary/ingress-canary-pbd8k" Sep 07 10:11:52.598618 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.598573 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp96\" (UniqueName: \"kubernetes.io/projected/7a78daa9-fda7-4dbc-9465-17e251db07f6-kube-api-access-tzp96\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:52.598736 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.598716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqjh\" (UniqueName: \"kubernetes.io/projected/9f1d9389-f077-4289-92f0-19197641f90c-kube-api-access-frqjh\") pod \"ingress-canary-pbd8k\" (UID: \"9f1d9389-f077-4289-92f0-19197641f90c\") " pod="openshift-ingress-canary/ingress-canary-pbd8k" Sep 07 10:11:52.600880 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.600860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-bound-sa-token\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.601072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.601049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjp8\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-kube-api-access-kkjp8\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:52.685733 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.685700 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pbd8k" Sep 07 10:11:52.973326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:52.973130 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pbd8k"] Sep 07 10:11:53.095106 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:53.094472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:53.095106 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:53.094530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:53.095106 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:53.094629 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 10:11:53.095106 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:53.094648 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 10:11:53.095106 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:53.094655 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9bbc69b-rrt8w: secret "image-registry-tls" not found Sep 07 10:11:53.095106 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:53.094707 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls podName:7a78daa9-fda7-4dbc-9465-17e251db07f6 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:54.094689465 +0000 UTC m=+36.080969780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls") pod "dns-default-bfpfk" (UID: "7a78daa9-fda7-4dbc-9465-17e251db07f6") : secret "dns-default-metrics-tls" not found Sep 07 10:11:53.095106 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:53.094722 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls podName:4496a2f9-ed1d-4e70-919d-b4968349702f nodeName:}" failed. No retries permitted until 2026-09-07 10:11:54.094716275 +0000 UTC m=+36.080996587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls") pod "image-registry-78f9bbc69b-rrt8w" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f") : secret "image-registry-tls" not found Sep 07 10:11:54.102699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:54.102663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:54.103139 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:54.102800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:54.103139 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:54.102825 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 10:11:54.103139 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:54.102894 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls podName:7a78daa9-fda7-4dbc-9465-17e251db07f6 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:56.102874358 +0000 UTC m=+38.089154686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls") pod "dns-default-bfpfk" (UID: "7a78daa9-fda7-4dbc-9465-17e251db07f6") : secret "dns-default-metrics-tls" not found Sep 07 10:11:54.103139 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:54.102898 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 10:11:54.103139 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:54.102909 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9bbc69b-rrt8w: secret "image-registry-tls" not found Sep 07 10:11:54.103139 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:54.102955 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls podName:4496a2f9-ed1d-4e70-919d-b4968349702f nodeName:}" failed. No retries permitted until 2026-09-07 10:11:56.102944208 +0000 UTC m=+38.089224523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls") pod "image-registry-78f9bbc69b-rrt8w" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f") : secret "image-registry-tls" not found Sep 07 10:11:54.271013 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:11:54.270979 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1d9389_f077_4289_92f0_19197641f90c.slice/crio-715c84711be4c4556fb6b0779e417ce8e2e4a4d5ee18461926944aa668195149 WatchSource:0}: Error finding container 715c84711be4c4556fb6b0779e417ce8e2e4a4d5ee18461926944aa668195149: Status 404 returned error can't find the container with id 715c84711be4c4556fb6b0779e417ce8e2e4a4d5ee18461926944aa668195149 Sep 07 10:11:54.831798 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:54.831763 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f7afb78-4695-4bf1-9835-117132b6b761" containerID="7b6255f349c1586cffbc22213cf83972b2fef5176fe15af5b9a6d6bf22e519ee" exitCode=0 Sep 07 10:11:54.831980 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:54.831837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" event={"ID":"1f7afb78-4695-4bf1-9835-117132b6b761","Type":"ContainerDied","Data":"7b6255f349c1586cffbc22213cf83972b2fef5176fe15af5b9a6d6bf22e519ee"} Sep 07 10:11:54.833095 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:54.833070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pbd8k" event={"ID":"9f1d9389-f077-4289-92f0-19197641f90c","Type":"ContainerStarted","Data":"715c84711be4c4556fb6b0779e417ce8e2e4a4d5ee18461926944aa668195149"} Sep 07 10:11:55.838466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:55.838428 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f7afb78-4695-4bf1-9835-117132b6b761" containerID="07fe9d4e9d1060a285d1ccdd539733912f4c2ee571839b68ecc2bae0e5660ab1" exitCode=0 Sep 07 10:11:55.839461 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:55.838486 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" event={"ID":"1f7afb78-4695-4bf1-9835-117132b6b761","Type":"ContainerDied","Data":"07fe9d4e9d1060a285d1ccdd539733912f4c2ee571839b68ecc2bae0e5660ab1"} Sep 07 10:11:56.118611 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:56.118530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:11:56.118611 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:56.118580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:11:56.118819 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:56.118695 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 10:11:56.118819 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:56.118711 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 10:11:56.118819 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:56.118717 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9bbc69b-rrt8w: secret "image-registry-tls" not found Sep 07 10:11:56.118819 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:56.118781 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls podName:7a78daa9-fda7-4dbc-9465-17e251db07f6 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:00.118763131 +0000 UTC m=+42.105043454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls") pod "dns-default-bfpfk" (UID: "7a78daa9-fda7-4dbc-9465-17e251db07f6") : secret "dns-default-metrics-tls" not found Sep 07 10:11:56.118819 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:11:56.118799 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls podName:4496a2f9-ed1d-4e70-919d-b4968349702f nodeName:}" failed. No retries permitted until 2026-09-07 10:12:00.118789797 +0000 UTC m=+42.105070110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls") pod "image-registry-78f9bbc69b-rrt8w" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f") : secret "image-registry-tls" not found Sep 07 10:11:56.844038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:56.844000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" event={"ID":"1f7afb78-4695-4bf1-9835-117132b6b761","Type":"ContainerStarted","Data":"5769953df33916cacb873c4773dbbe355aae31638f81894be371c191e02fd33b"} Sep 07 10:11:56.868000 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:56.867949 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5c5vj" podStartSLOduration=9.436564898 podStartE2EDuration="38.867932944s" totalImagesPullingTime="29.431368046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:56.86741466 +0000 UTC m=+38.853694999" watchObservedRunningTime="2026-09-07 10:11:56.867932944 +0000 UTC m=+38.854213278" Sep 07 10:11:57.847833 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:57.847656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pbd8k" event={"ID":"9f1d9389-f077-4289-92f0-19197641f90c","Type":"ContainerStarted","Data":"881e0c6c39474e412b38fd5ae8fb7ed70bc79029605c4b38950bc39cc5c0da4c"} Sep 07 10:11:57.864125 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:11:57.864082 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pbd8k" podStartSLOduration=3.1678488910000002 podStartE2EDuration="5.864066674s" totalImagesPullingTime="2.696217783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:57.86395008 +0000 UTC m=+39.850230414" watchObservedRunningTime="2026-09-07 10:11:57.864066674 +0000 UTC m=+39.850347009" Sep 07 10:12:00.149915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:00.149871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:12:00.149915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:00.149922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:12:00.150427 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:00.150028 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 10:12:00.150427 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:00.150053 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9bbc69b-rrt8w: secret "image-registry-tls" not found Sep 07 10:12:00.150427 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:00.150099 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls podName:4496a2f9-ed1d-4e70-919d-b4968349702f nodeName:}" failed. No retries permitted until 2026-09-07 10:12:08.150085034 +0000 UTC m=+50.136365346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls") pod "image-registry-78f9bbc69b-rrt8w" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f") : secret "image-registry-tls" not found Sep 07 10:12:00.150427 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:00.150030 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 10:12:00.150427 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:00.150261 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls podName:7a78daa9-fda7-4dbc-9465-17e251db07f6 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:08.150243842 +0000 UTC m=+50.136524168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls") pod "dns-default-bfpfk" (UID: "7a78daa9-fda7-4dbc-9465-17e251db07f6") : secret "dns-default-metrics-tls" not found Sep 07 10:12:04.999721 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:04.999688 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg"] Sep 07 10:12:05.034380 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.034343 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg"] Sep 07 10:12:05.034546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.034458 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.037170 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.037123 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:12:05.037170 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.037127 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 07 10:12:05.037348 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.037128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nsxm7\"" Sep 07 10:12:05.037348 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.037175 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 07 10:12:05.041082 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.039030 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 07 10:12:05.106298 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.106262 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h"] Sep 07 10:12:05.123321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.123296 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb"] Sep 07 10:12:05.123491 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.123446 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.126830 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.126804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ng9bp\"" Sep 07 10:12:05.126965 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.126819 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 07 10:12:05.126965 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.126828 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:12:05.127283 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.127217 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 07 10:12:05.127283 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.127274 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 07 10:12:05.149395 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.149370 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qkcbp"] Sep 07 10:12:05.149512 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.149446 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb" Sep 07 10:12:05.151947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.151925 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:12:05.151947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.151944 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 07 10:12:05.152134 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.152009 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hh9qf\"" Sep 07 10:12:05.170585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.170560 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw"] Sep 07 10:12:05.170719 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.170703 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.172999 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.172977 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 07 10:12:05.173090 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.173026 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 07 10:12:05.173150 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.173094 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 07 10:12:05.173242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.173215 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 07 10:12:05.173322 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.173305 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hfbfc\"" Sep 07 10:12:05.177983 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.177966 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 07 10:12:05.189017 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.188988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqhm\" (UniqueName: \"kubernetes.io/projected/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-kube-api-access-xqqhm\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.189112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.189029 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-serving-cert\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.189112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.189055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-config\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.190744 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.190729 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql"] Sep 07 10:12:05.190841 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.190829 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.193276 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.193252 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 07 10:12:05.193363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.193349 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 07 10:12:05.193407 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.193392 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xkndp\"" Sep 07 10:12:05.194742 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.194722 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 07 10:12:05.194808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.194758 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 07 10:12:05.216821 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.216801 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb"] Sep 07 10:12:05.217038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.216839 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h"] Sep 07 10:12:05.217038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.216863 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qkcbp"] Sep 07 10:12:05.217038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.216882 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw"] Sep 07 10:12:05.217038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.216900 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql"] Sep 07 10:12:05.217038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.216930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:05.217038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.216944 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nbkdc"] Sep 07 10:12:05.219389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.219371 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9xwvz\"" Sep 07 10:12:05.219595 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.219568 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:12:05.219683 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.219596 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 07 10:12:05.219868 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.219853 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 07 10:12:05.232861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.232789 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nbkdc"] Sep 07 10:12:05.233008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.232866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.235447 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.235412 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 07 10:12:05.235552 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.235534 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 07 10:12:05.235646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.235575 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vq2qs\"" Sep 07 10:12:05.235646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.235610 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:12:05.235646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.235634 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 07 10:12:05.240311 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.240294 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 07 10:12:05.290155 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqhm\" (UniqueName: \"kubernetes.io/projected/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-kube-api-access-xqqhm\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.290155 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t57n7\" (UniqueName: \"kubernetes.io/projected/be97b92a-7c7c-4cad-a5be-07bd910af233-kube-api-access-t57n7\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.290368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/778d7208-c6a5-45c3-84d5-618bb54a7924-snapshots\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.290368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.290368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk94h\" (UniqueName: \"kubernetes.io/projected/778d7208-c6a5-45c3-84d5-618bb54a7924-kube-api-access-wk94h\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.290368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/778d7208-c6a5-45c3-84d5-618bb54a7924-serving-cert\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.290368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778d7208-c6a5-45c3-84d5-618bb54a7924-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.290368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778d7208-c6a5-45c3-84d5-618bb54a7924-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.290689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.290689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-serving-cert\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.290689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-config\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.290689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/778d7208-c6a5-45c3-84d5-618bb54a7924-tmp\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.290689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ltz\" (UniqueName: \"kubernetes.io/projected/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-kube-api-access-42ltz\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.290689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be97b92a-7c7c-4cad-a5be-07bd910af233-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.290689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be97b92a-7c7c-4cad-a5be-07bd910af233-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.290689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9h6n\" (UniqueName: \"kubernetes.io/projected/e187bc78-fd0c-4568-a569-788cf31a0982-kube-api-access-x9h6n\") pod \"volume-data-source-validator-695c895bd6-kvrrb\" (UID: \"e187bc78-fd0c-4568-a569-788cf31a0982\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb" Sep 07 10:12:05.290976 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.290960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-config\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.294008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.293989 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-serving-cert\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.299250 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.299231 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-jm9hj"] Sep 07 10:12:05.301771 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.301751 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqhm\" (UniqueName: \"kubernetes.io/projected/829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b-kube-api-access-xqqhm\") pod \"service-ca-operator-859f94c4b-n24qg\" (UID: \"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.318967 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.318948 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-l922z"] Sep 07 10:12:05.319086 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.319071 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-jm9hj" Sep 07 10:12:05.321787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.321664 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vgps7\"" Sep 07 10:12:05.341137 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.341114 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6bd85f49dd-l2v9q"] Sep 07 10:12:05.341274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.341258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:05.343691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.343670 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 07 10:12:05.343804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.343732 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 07 10:12:05.343804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.343765 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7472n\"" Sep 07 10:12:05.345633 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.345617 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" Sep 07 10:12:05.359725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.359699 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-jm9hj"] Sep 07 10:12:05.359725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.359726 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-l922z"] Sep 07 10:12:05.359922 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.359745 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bd85f49dd-l2v9q"] Sep 07 10:12:05.359922 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.359824 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.362178 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.362143 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 07 10:12:05.362508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.362407 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 07 10:12:05.362508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.362456 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 07 10:12:05.362508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.362488 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 07 10:12:05.362740 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.362491 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 07 10:12:05.362740 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.362524 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-g7ptg\"" Sep 07 10:12:05.362838 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.362786 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 07 10:12:05.391683 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.391654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57n7\" (UniqueName: \"kubernetes.io/projected/be97b92a-7c7c-4cad-a5be-07bd910af233-kube-api-access-t57n7\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.391831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.391690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/778d7208-c6a5-45c3-84d5-618bb54a7924-snapshots\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.391831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.391708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.391831 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.391804 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:05.391975 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.391866 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls podName:6c6b7ec1-a764-41f0-8541-a7b52c6989a2 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:05.891847291 +0000 UTC m=+47.878127604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wzscw" (UID: "6c6b7ec1-a764-41f0-8541-a7b52c6989a2") : secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:05.391975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.391891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1749d01c-371e-42dc-9c83-d31ecb1b2788-serving-cert\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.391975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.391940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk94h\" (UniqueName: \"kubernetes.io/projected/778d7208-c6a5-45c3-84d5-618bb54a7924-kube-api-access-wk94h\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.391975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.391970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/778d7208-c6a5-45c3-84d5-618bb54a7924-serving-cert\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.392225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8gxr\" (UniqueName: \"kubernetes.io/projected/7fdc76fd-992d-420d-a565-e78ab94703c0-kube-api-access-t8gxr\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:05.392225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778d7208-c6a5-45c3-84d5-618bb54a7924-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.392225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778d7208-c6a5-45c3-84d5-618bb54a7924-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.392225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.392405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1749d01c-371e-42dc-9c83-d31ecb1b2788-config\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.392405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hr86\" (UniqueName: \"kubernetes.io/projected/1749d01c-371e-42dc-9c83-d31ecb1b2788-kube-api-access-4hr86\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.392405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/778d7208-c6a5-45c3-84d5-618bb54a7924-tmp\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.392405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ltz\" (UniqueName: \"kubernetes.io/projected/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-kube-api-access-42ltz\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.392405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be97b92a-7c7c-4cad-a5be-07bd910af233-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.392405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be97b92a-7c7c-4cad-a5be-07bd910af233-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.392405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392392 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/778d7208-c6a5-45c3-84d5-618bb54a7924-snapshots\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.392723 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392406 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9h6n\" (UniqueName: \"kubernetes.io/projected/e187bc78-fd0c-4568-a569-788cf31a0982-kube-api-access-x9h6n\") pod \"volume-data-source-validator-695c895bd6-kvrrb\" (UID: \"e187bc78-fd0c-4568-a569-788cf31a0982\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb" Sep 07 10:12:05.392723 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392467 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:05.392723 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392498 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1749d01c-371e-42dc-9c83-d31ecb1b2788-trusted-ca\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.393029 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392864 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778d7208-c6a5-45c3-84d5-618bb54a7924-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.393029 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392963 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/778d7208-c6a5-45c3-84d5-618bb54a7924-tmp\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.393029 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.392996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778d7208-c6a5-45c3-84d5-618bb54a7924-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.393229 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.393101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be97b92a-7c7c-4cad-a5be-07bd910af233-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.393297 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.393269 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.394622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.394601 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/778d7208-c6a5-45c3-84d5-618bb54a7924-serving-cert\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.395278 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.395240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be97b92a-7c7c-4cad-a5be-07bd910af233-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.400912 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.400889 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57n7\" (UniqueName: \"kubernetes.io/projected/be97b92a-7c7c-4cad-a5be-07bd910af233-kube-api-access-t57n7\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8c94h\" (UID: \"be97b92a-7c7c-4cad-a5be-07bd910af233\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.401076 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.401058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk94h\" (UniqueName: \"kubernetes.io/projected/778d7208-c6a5-45c3-84d5-618bb54a7924-kube-api-access-wk94h\") pod \"insights-operator-76bcd6ccf5-qkcbp\" (UID: \"778d7208-c6a5-45c3-84d5-618bb54a7924\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.401454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.401406 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ltz\" (UniqueName: \"kubernetes.io/projected/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-kube-api-access-42ltz\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.402349 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.402329 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9h6n\" (UniqueName: \"kubernetes.io/projected/e187bc78-fd0c-4568-a569-788cf31a0982-kube-api-access-x9h6n\") pod \"volume-data-source-validator-695c895bd6-kvrrb\" (UID: \"e187bc78-fd0c-4568-a569-788cf31a0982\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb" Sep 07 10:12:05.433495 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.433466 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" Sep 07 10:12:05.457823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.457791 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb" Sep 07 10:12:05.478771 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.478745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" Sep 07 10:12:05.493069 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1749d01c-371e-42dc-9c83-d31ecb1b2788-trusted-ca\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.493184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-stats-auth\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.493184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1749d01c-371e-42dc-9c83-d31ecb1b2788-serving-cert\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.493184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.493335 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493200 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-default-certificate\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.493335 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8gxr\" (UniqueName: \"kubernetes.io/projected/7fdc76fd-992d-420d-a565-e78ab94703c0-kube-api-access-t8gxr\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:05.493428 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d477d2b-4f2b-41ee-9800-73c0650da064-nginx-conf\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:05.493482 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:05.493482 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1749d01c-371e-42dc-9c83-d31ecb1b2788-config\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.493594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hr86\" (UniqueName: \"kubernetes.io/projected/1749d01c-371e-42dc-9c83-d31ecb1b2788-kube-api-access-4hr86\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.493594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.493594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qng\" (UniqueName: \"kubernetes.io/projected/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-kube-api-access-h6qng\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.493748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:05.493748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.493651 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w628d\" (UniqueName: \"kubernetes.io/projected/59986d9c-bf98-4061-bf08-682e22c5c834-kube-api-access-w628d\") pod \"network-check-source-685456655f-jm9hj\" (UID: \"59986d9c-bf98-4061-bf08-682e22c5c834\") " pod="openshift-network-diagnostics/network-check-source-685456655f-jm9hj" Sep 07 10:12:05.493748 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.493733 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 10:12:05.493920 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.493790 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls podName:7fdc76fd-992d-420d-a565-e78ab94703c0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:05.993772599 +0000 UTC m=+47.980052950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sh6ql" (UID: "7fdc76fd-992d-420d-a565-e78ab94703c0") : secret "samples-operator-tls" not found Sep 07 10:12:05.494121 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.494031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1749d01c-371e-42dc-9c83-d31ecb1b2788-config\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.494121 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.494097 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1749d01c-371e-42dc-9c83-d31ecb1b2788-trusted-ca\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.495649 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.495631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1749d01c-371e-42dc-9c83-d31ecb1b2788-serving-cert\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.501752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.501725 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8gxr\" (UniqueName: \"kubernetes.io/projected/7fdc76fd-992d-420d-a565-e78ab94703c0-kube-api-access-t8gxr\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:05.502218 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.502201 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hr86\" (UniqueName: \"kubernetes.io/projected/1749d01c-371e-42dc-9c83-d31ecb1b2788-kube-api-access-4hr86\") pod \"console-operator-85b454874b-nbkdc\" (UID: \"1749d01c-371e-42dc-9c83-d31ecb1b2788\") " pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.541266 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.541198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:05.594132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.594088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qng\" (UniqueName: \"kubernetes.io/projected/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-kube-api-access-h6qng\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.594270 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.594181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w628d\" (UniqueName: \"kubernetes.io/projected/59986d9c-bf98-4061-bf08-682e22c5c834-kube-api-access-w628d\") pod \"network-check-source-685456655f-jm9hj\" (UID: \"59986d9c-bf98-4061-bf08-682e22c5c834\") " pod="openshift-network-diagnostics/network-check-source-685456655f-jm9hj" Sep 07 10:12:05.594270 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.594235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-stats-auth\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.594387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.594303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.594387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.594330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-default-certificate\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.594387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.594382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d477d2b-4f2b-41ee-9800-73c0650da064-nginx-conf\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:05.594600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.594426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:05.594600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.594464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.595044 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.594602 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 10:12:05.595044 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.594672 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:06.094651534 +0000 UTC m=+48.080931873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : secret "router-metrics-certs-default" not found Sep 07 10:12:05.595044 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.594872 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:06.094852979 +0000 UTC m=+48.081133313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : configmap references non-existent config key: service-ca.crt Sep 07 10:12:05.595437 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.595359 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 10:12:05.595437 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.595412 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert podName:1d477d2b-4f2b-41ee-9800-73c0650da064 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:06.09539698 +0000 UTC m=+48.081677294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-l922z" (UID: "1d477d2b-4f2b-41ee-9800-73c0650da064") : secret "networking-console-plugin-cert" not found Sep 07 10:12:05.595941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.595921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d477d2b-4f2b-41ee-9800-73c0650da064-nginx-conf\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:05.597516 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.597492 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-default-certificate\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.597602 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.597529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-stats-auth\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.605586 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.605540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w628d\" (UniqueName: \"kubernetes.io/projected/59986d9c-bf98-4061-bf08-682e22c5c834-kube-api-access-w628d\") pod \"network-check-source-685456655f-jm9hj\" (UID: \"59986d9c-bf98-4061-bf08-682e22c5c834\") " pod="openshift-network-diagnostics/network-check-source-685456655f-jm9hj" Sep 07 10:12:05.605666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.605641 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qng\" (UniqueName: \"kubernetes.io/projected/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-kube-api-access-h6qng\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:05.629362 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.629317 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-jm9hj" Sep 07 10:12:05.631688 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:05.631665 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829ad1ad_70d6_4f45_8dfc_e6428e2c5b5b.slice/crio-16aec0a940549baf29c354f03dc62a1182ba35a197177a123e72986b2315b6ed WatchSource:0}: Error finding container 16aec0a940549baf29c354f03dc62a1182ba35a197177a123e72986b2315b6ed: Status 404 returned error can't find the container with id 16aec0a940549baf29c354f03dc62a1182ba35a197177a123e72986b2315b6ed Sep 07 10:12:05.636923 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.636855 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg"] Sep 07 10:12:05.741088 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.740679 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h"] Sep 07 10:12:05.742358 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:05.742328 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe97b92a_7c7c_4cad_a5be_07bd910af233.slice/crio-9a0f8fc3c902cf365228f552c6df2685fc88302e686a575c318383f0ac365a6d WatchSource:0}: Error finding container 9a0f8fc3c902cf365228f552c6df2685fc88302e686a575c318383f0ac365a6d: Status 404 returned error can't find the container with id 9a0f8fc3c902cf365228f552c6df2685fc88302e686a575c318383f0ac365a6d Sep 07 10:12:05.772925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.772898 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb"] Sep 07 10:12:05.775820 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:05.775800 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode187bc78_fd0c_4568_a569_788cf31a0982.slice/crio-b58555b793eda87a928c89f3a1d88d6fdb8d4fa1133993a5f785f9067787191d WatchSource:0}: Error finding container b58555b793eda87a928c89f3a1d88d6fdb8d4fa1133993a5f785f9067787191d: Status 404 returned error can't find the container with id b58555b793eda87a928c89f3a1d88d6fdb8d4fa1133993a5f785f9067787191d Sep 07 10:12:05.796493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.796439 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qkcbp"] Sep 07 10:12:05.798766 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:05.798749 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778d7208_c6a5_45c3_84d5_618bb54a7924.slice/crio-68f7fed6e087a9d7585cabde4f340b30078168599da850960ecbdc98ce775408 WatchSource:0}: Error finding container 68f7fed6e087a9d7585cabde4f340b30078168599da850960ecbdc98ce775408: Status 404 returned error can't find the container with id 68f7fed6e087a9d7585cabde4f340b30078168599da850960ecbdc98ce775408 Sep 07 10:12:05.839666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.839637 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nbkdc"] Sep 07 10:12:05.843705 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:05.843681 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1749d01c_371e_42dc_9c83_d31ecb1b2788.slice/crio-7901bcc768a4cda2f6c9063dae93386a636149dbf1b904b0b7135ee4860bc832 WatchSource:0}: Error finding container 7901bcc768a4cda2f6c9063dae93386a636149dbf1b904b0b7135ee4860bc832: Status 404 returned error can't find the container with id 7901bcc768a4cda2f6c9063dae93386a636149dbf1b904b0b7135ee4860bc832 Sep 07 10:12:05.866086 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.866055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" event={"ID":"1749d01c-371e-42dc-9c83-d31ecb1b2788","Type":"ContainerStarted","Data":"7901bcc768a4cda2f6c9063dae93386a636149dbf1b904b0b7135ee4860bc832"} Sep 07 10:12:05.867173 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.867139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" event={"ID":"778d7208-c6a5-45c3-84d5-618bb54a7924","Type":"ContainerStarted","Data":"68f7fed6e087a9d7585cabde4f340b30078168599da850960ecbdc98ce775408"} Sep 07 10:12:05.868201 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.868175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" event={"ID":"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b","Type":"ContainerStarted","Data":"16aec0a940549baf29c354f03dc62a1182ba35a197177a123e72986b2315b6ed"} Sep 07 10:12:05.869148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.869128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb" event={"ID":"e187bc78-fd0c-4568-a569-788cf31a0982","Type":"ContainerStarted","Data":"b58555b793eda87a928c89f3a1d88d6fdb8d4fa1133993a5f785f9067787191d"} Sep 07 10:12:05.870213 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.870183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" event={"ID":"be97b92a-7c7c-4cad-a5be-07bd910af233","Type":"ContainerStarted","Data":"9a0f8fc3c902cf365228f552c6df2685fc88302e686a575c318383f0ac365a6d"} Sep 07 10:12:05.897724 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.897686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:05.897866 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.897836 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:05.897919 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.897905 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls podName:6c6b7ec1-a764-41f0-8541-a7b52c6989a2 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:06.897888681 +0000 UTC m=+48.884169000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wzscw" (UID: "6c6b7ec1-a764-41f0-8541-a7b52c6989a2") : secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:05.922643 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.922617 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-jm9hj"] Sep 07 10:12:05.925246 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:05.925221 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59986d9c_bf98_4061_bf08_682e22c5c834.slice/crio-36322711a2997900425dddcc36081854995b2419d0aa558f5c619c0f2f420b4a WatchSource:0}: Error finding container 36322711a2997900425dddcc36081854995b2419d0aa558f5c619c0f2f420b4a: Status 404 returned error can't find the container with id 36322711a2997900425dddcc36081854995b2419d0aa558f5c619c0f2f420b4a Sep 07 10:12:05.999018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:05.998986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:05.999180 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.999137 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 10:12:05.999245 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:05.999220 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls podName:7fdc76fd-992d-420d-a565-e78ab94703c0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:06.999203721 +0000 UTC m=+48.985484033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sh6ql" (UID: "7fdc76fd-992d-420d-a565-e78ab94703c0") : secret "samples-operator-tls" not found Sep 07 10:12:06.100145 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:06.100113 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:06.100542 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:06.100289 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 10:12:06.100542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:06.100311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:06.100542 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:06.100365 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:07.100343353 +0000 UTC m=+49.086623688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : secret "router-metrics-certs-default" not found Sep 07 10:12:06.100542 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:06.100433 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:07.100416777 +0000 UTC m=+49.086697106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : configmap references non-existent config key: service-ca.crt Sep 07 10:12:06.100542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:06.100456 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:06.100755 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:06.100566 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 10:12:06.100755 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:06.100618 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert podName:1d477d2b-4f2b-41ee-9800-73c0650da064 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:07.100605365 +0000 UTC m=+49.086885691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-l922z" (UID: "1d477d2b-4f2b-41ee-9800-73c0650da064") : secret "networking-console-plugin-cert" not found Sep 07 10:12:06.875948 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:06.875871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-jm9hj" event={"ID":"59986d9c-bf98-4061-bf08-682e22c5c834","Type":"ContainerStarted","Data":"36322711a2997900425dddcc36081854995b2419d0aa558f5c619c0f2f420b4a"} Sep 07 10:12:06.909309 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:06.909268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:06.909559 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:06.909542 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:06.909630 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:06.909610 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls podName:6c6b7ec1-a764-41f0-8541-a7b52c6989a2 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:08.909591666 +0000 UTC m=+50.895871993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wzscw" (UID: "6c6b7ec1-a764-41f0-8541-a7b52c6989a2") : secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:07.010274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:07.010230 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:07.010449 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:07.010395 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 10:12:07.010512 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:07.010464 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls podName:7fdc76fd-992d-420d-a565-e78ab94703c0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:09.010444013 +0000 UTC m=+50.996724344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sh6ql" (UID: "7fdc76fd-992d-420d-a565-e78ab94703c0") : secret "samples-operator-tls" not found Sep 07 10:12:07.112312 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:07.111412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:07.112312 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:07.111506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:07.112312 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:07.111549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:07.112312 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:07.111677 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 10:12:07.112312 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:07.111742 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:09.111721101 +0000 UTC m=+51.098001415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : secret "router-metrics-certs-default" not found Sep 07 10:12:07.112312 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:07.112131 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:09.112113956 +0000 UTC m=+51.098394287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : configmap references non-existent config key: service-ca.crt Sep 07 10:12:07.112312 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:07.112234 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 10:12:07.112312 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:07.112273 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert podName:1d477d2b-4f2b-41ee-9800-73c0650da064 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:09.112261067 +0000 UTC m=+51.098541383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-l922z" (UID: "1d477d2b-4f2b-41ee-9800-73c0650da064") : secret "networking-console-plugin-cert" not found Sep 07 10:12:07.882087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:07.881977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb" event={"ID":"e187bc78-fd0c-4568-a569-788cf31a0982","Type":"ContainerStarted","Data":"322ad23b14561594b00fb8bc20a37177de7f5e7967df805008883b4fee8db582"} Sep 07 10:12:08.222667 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:08.222631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:12:08.223143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:08.222759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:12:08.223143 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:08.222905 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 10:12:08.223143 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:08.222965 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls podName:7a78daa9-fda7-4dbc-9465-17e251db07f6 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:24.222945905 +0000 UTC m=+66.209226236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls") pod "dns-default-bfpfk" (UID: "7a78daa9-fda7-4dbc-9465-17e251db07f6") : secret "dns-default-metrics-tls" not found Sep 07 10:12:08.223332 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:08.223288 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 10:12:08.223332 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:08.223306 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9bbc69b-rrt8w: secret "image-registry-tls" not found Sep 07 10:12:08.223449 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:08.223353 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls podName:4496a2f9-ed1d-4e70-919d-b4968349702f nodeName:}" failed. No retries permitted until 2026-09-07 10:12:24.223340787 +0000 UTC m=+66.209621099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls") pod "image-registry-78f9bbc69b-rrt8w" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f") : secret "image-registry-tls" not found Sep 07 10:12:08.932196 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:08.932143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:08.932391 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:08.932256 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:08.932391 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:08.932315 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls podName:6c6b7ec1-a764-41f0-8541-a7b52c6989a2 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:12.932300158 +0000 UTC m=+54.918580469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wzscw" (UID: "6c6b7ec1-a764-41f0-8541-a7b52c6989a2") : secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:09.032999 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:09.032952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:09.033221 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:09.033201 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 10:12:09.033293 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:09.033275 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls podName:7fdc76fd-992d-420d-a565-e78ab94703c0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:13.033254671 +0000 UTC m=+55.019535006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sh6ql" (UID: "7fdc76fd-992d-420d-a565-e78ab94703c0") : secret "samples-operator-tls" not found Sep 07 10:12:09.133898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:09.133867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:09.134050 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:09.133910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:09.134050 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:09.134000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:09.134050 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:09.134011 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 10:12:09.134244 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:09.134068 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert podName:1d477d2b-4f2b-41ee-9800-73c0650da064 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:13.134053456 +0000 UTC m=+55.120333783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-l922z" (UID: "1d477d2b-4f2b-41ee-9800-73c0650da064") : secret "networking-console-plugin-cert" not found Sep 07 10:12:09.134244 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:09.134117 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:13.134105586 +0000 UTC m=+55.120385919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : configmap references non-existent config key: service-ca.crt Sep 07 10:12:09.134244 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:09.134141 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 10:12:09.134244 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:09.134227 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:13.134208403 +0000 UTC m=+55.120488715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : secret "router-metrics-certs-default" not found Sep 07 10:12:11.896744 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.896711 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/0.log" Sep 07 10:12:11.897222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.896761 2565 generic.go:356] "Generic (PLEG): container finished" podID="1749d01c-371e-42dc-9c83-d31ecb1b2788" containerID="c45dbfc9b4cdbd4632f0314903619a5869e0c55a03532ba5cc4bd2d0bca1b550" exitCode=255 Sep 07 10:12:11.897222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.896862 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" event={"ID":"1749d01c-371e-42dc-9c83-d31ecb1b2788","Type":"ContainerDied","Data":"c45dbfc9b4cdbd4632f0314903619a5869e0c55a03532ba5cc4bd2d0bca1b550"} Sep 07 10:12:11.897222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.897078 2565 scope.go:117] "RemoveContainer" containerID="c45dbfc9b4cdbd4632f0314903619a5869e0c55a03532ba5cc4bd2d0bca1b550" Sep 07 10:12:11.898664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.898621 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-jm9hj" event={"ID":"59986d9c-bf98-4061-bf08-682e22c5c834","Type":"ContainerStarted","Data":"aa663b57f9f1a341f70861beae48c9fd3baceb6a89ff0c73ebd3d51ed8baac1e"} Sep 07 10:12:11.900802 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.900646 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" event={"ID":"778d7208-c6a5-45c3-84d5-618bb54a7924","Type":"ContainerStarted","Data":"8a198b61fd05eab08c3daae3601881aab8a3bd055fc8b752eece034ccdf579ff"} Sep 07 10:12:11.904935 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.904909 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" event={"ID":"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b","Type":"ContainerStarted","Data":"06f26fb4a674029e8fe234e323dcff857a41a24d9b1beb049b2320fa6c224b0d"} Sep 07 10:12:11.907048 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.906998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" event={"ID":"be97b92a-7c7c-4cad-a5be-07bd910af233","Type":"ContainerStarted","Data":"66e3c3f1b36d9dee21e9ffea9b941910919bbfc8a754c757d5e26a09224d850d"} Sep 07 10:12:11.914152 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.914113 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-kvrrb" podStartSLOduration=5.058800417 podStartE2EDuration="6.914100928s" totalImagesPullingTime="1.855300511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:07.89884985 +0000 UTC m=+49.885130188" watchObservedRunningTime="2026-09-07 10:12:11.914100928 +0000 UTC m=+53.900381261" Sep 07 10:12:11.929105 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.929065 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" podStartSLOduration=1.225110572 podStartE2EDuration="6.929048839s" totalImagesPullingTime="5.703938267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:11.928617696 +0000 UTC m=+53.914898030" watchObservedRunningTime="2026-09-07 10:12:11.929048839 +0000 UTC m=+53.915329174" Sep 07 10:12:11.972725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.970312 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" podStartSLOduration=2.153601531 podStartE2EDuration="7.970292081s" totalImagesPullingTime="5.81669055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:11.969245705 +0000 UTC m=+53.955526069" watchObservedRunningTime="2026-09-07 10:12:11.970292081 +0000 UTC m=+53.956572419" Sep 07 10:12:11.972725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:11.970431 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" podStartSLOduration=1.326714421 podStartE2EDuration="6.970422969s" totalImagesPullingTime="5.643708548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:11.946832609 +0000 UTC m=+53.933112947" watchObservedRunningTime="2026-09-07 10:12:11.970422969 +0000 UTC m=+53.956703304" Sep 07 10:12:12.912607 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:12.912580 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:12:12.913062 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:12.913048 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/0.log" Sep 07 10:12:12.913114 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:12.913083 2565 generic.go:356] "Generic (PLEG): container finished" podID="1749d01c-371e-42dc-9c83-d31ecb1b2788" containerID="4812cb1e5d3e3f9ee4ebf439833608c9a8cd523cdbf6462fc4a364c3acef1952" exitCode=255 Sep 07 10:12:12.913244 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:12.913213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" event={"ID":"1749d01c-371e-42dc-9c83-d31ecb1b2788","Type":"ContainerDied","Data":"4812cb1e5d3e3f9ee4ebf439833608c9a8cd523cdbf6462fc4a364c3acef1952"} Sep 07 10:12:12.913337 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:12.913305 2565 scope.go:117] "RemoveContainer" containerID="c45dbfc9b4cdbd4632f0314903619a5869e0c55a03532ba5cc4bd2d0bca1b550" Sep 07 10:12:12.913440 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:12.913420 2565 scope.go:117] "RemoveContainer" containerID="4812cb1e5d3e3f9ee4ebf439833608c9a8cd523cdbf6462fc4a364c3acef1952" Sep 07 10:12:12.913651 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:12.913619 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nbkdc_openshift-console-operator(1749d01c-371e-42dc-9c83-d31ecb1b2788)\"" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" podUID="1749d01c-371e-42dc-9c83-d31ecb1b2788" Sep 07 10:12:12.929603 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:12.929551 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-jm9hj" podStartSLOduration=2.402039146 podStartE2EDuration="7.929524834s" totalImagesPullingTime="5.527485688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:11.986476281 +0000 UTC m=+53.972756608" watchObservedRunningTime="2026-09-07 10:12:12.929524834 +0000 UTC m=+54.915805170" Sep 07 10:12:12.973590 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:12.973553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:12.973798 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:12.973725 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:12.973862 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:12.973801 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls podName:6c6b7ec1-a764-41f0-8541-a7b52c6989a2 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:20.973780013 +0000 UTC m=+62.960060328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wzscw" (UID: "6c6b7ec1-a764-41f0-8541-a7b52c6989a2") : secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:13.074915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.074880 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:13.075077 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.075001 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 07 10:12:13.075077 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.075052 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls podName:7fdc76fd-992d-420d-a565-e78ab94703c0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.075038357 +0000 UTC m=+63.061318668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sh6ql" (UID: "7fdc76fd-992d-420d-a565-e78ab94703c0") : secret "samples-operator-tls" not found Sep 07 10:12:13.175816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.175722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:13.175816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.175799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:13.176011 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.175838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:13.176011 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.175905 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.175883866 +0000 UTC m=+63.162164195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : configmap references non-existent config key: service-ca.crt Sep 07 10:12:13.176011 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.175942 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 10:12:13.176011 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.175945 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 10:12:13.176011 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.175979 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.175965975 +0000 UTC m=+63.162246288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : secret "router-metrics-certs-default" not found Sep 07 10:12:13.176011 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.176004 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert podName:1d477d2b-4f2b-41ee-9800-73c0650da064 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.175990019 +0000 UTC m=+63.162270331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-l922z" (UID: "1d477d2b-4f2b-41ee-9800-73c0650da064") : secret "networking-console-plugin-cert" not found Sep 07 10:12:13.476598 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.476567 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rs2dm"] Sep 07 10:12:13.479787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.479765 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.482544 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.482520 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 07 10:12:13.482657 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.482541 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 07 10:12:13.482657 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.482623 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gp5c5\"" Sep 07 10:12:13.489247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.489219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rs2dm"] Sep 07 10:12:13.580072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.580044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.580264 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.580086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60e57c56-bb1b-42cd-a21c-90a1cae39c71-data-volume\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.580264 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.580123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.580264 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.580243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60e57c56-bb1b-42cd-a21c-90a1cae39c71-crio-socket\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.580364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.580325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfscw\" (UniqueName: \"kubernetes.io/projected/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-api-access-pfscw\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.681022 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.680990 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfscw\" (UniqueName: \"kubernetes.io/projected/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-api-access-pfscw\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.681224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.681047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.681224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.681078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60e57c56-bb1b-42cd-a21c-90a1cae39c71-data-volume\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.681224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.681106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.681224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.681147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60e57c56-bb1b-42cd-a21c-90a1cae39c71-crio-socket\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.681428 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.681257 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 10:12:13.681428 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.681267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60e57c56-bb1b-42cd-a21c-90a1cae39c71-crio-socket\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.681428 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.681263 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 10:12:13.681428 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.681335 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:14.181314719 +0000 UTC m=+56.167595040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : secret "insights-runtime-extractor-tls" not found Sep 07 10:12:13.681428 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.681358 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:14.18134885 +0000 UTC m=+56.167629167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : configmap "kube-rbac-proxy" not found Sep 07 10:12:13.681590 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.681491 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60e57c56-bb1b-42cd-a21c-90a1cae39c71-data-volume\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.690028 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.690001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfscw\" (UniqueName: \"kubernetes.io/projected/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-api-access-pfscw\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:13.918088 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.918013 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:12:13.918451 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:13.918360 2565 scope.go:117] "RemoveContainer" containerID="4812cb1e5d3e3f9ee4ebf439833608c9a8cd523cdbf6462fc4a364c3acef1952" Sep 07 10:12:13.918525 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:13.918506 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nbkdc_openshift-console-operator(1749d01c-371e-42dc-9c83-d31ecb1b2788)\"" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" podUID="1749d01c-371e-42dc-9c83-d31ecb1b2788" Sep 07 10:12:14.186456 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:14.186368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:14.186456 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:14.186424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:14.186641 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:14.186519 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 10:12:14.186641 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:14.186570 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 10:12:14.186641 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:14.186579 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.186563792 +0000 UTC m=+57.172844104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : secret "insights-runtime-extractor-tls" not found Sep 07 10:12:14.186641 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:14.186627 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.186613615 +0000 UTC m=+57.172893927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : configmap "kube-rbac-proxy" not found Sep 07 10:12:15.195787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:15.195743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:15.196336 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:15.195887 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 10:12:15.196336 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:15.195935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:15.196336 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:15.195951 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:17.195935805 +0000 UTC m=+59.182216116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : configmap "kube-rbac-proxy" not found Sep 07 10:12:15.196336 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:15.196037 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 10:12:15.196336 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:15.196096 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:17.196080569 +0000 UTC m=+59.182360901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : secret "insights-runtime-extractor-tls" not found Sep 07 10:12:15.541714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:15.541688 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:15.541888 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:15.541758 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:15.542089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:15.542072 2565 scope.go:117] "RemoveContainer" containerID="4812cb1e5d3e3f9ee4ebf439833608c9a8cd523cdbf6462fc4a364c3acef1952" Sep 07 10:12:15.542264 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:15.542245 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nbkdc_openshift-console-operator(1749d01c-371e-42dc-9c83-d31ecb1b2788)\"" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" podUID="1749d01c-371e-42dc-9c83-d31ecb1b2788" Sep 07 10:12:16.428546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:16.428521 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9qwl_154eb144-57c7-41b2-8ce3-d7efce4a447e/dns-node-resolver/0.log" Sep 07 10:12:17.214570 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:17.214531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:17.214771 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:17.214668 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 10:12:17.214771 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:17.214688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:17.214771 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:17.214742 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.214720354 +0000 UTC m=+63.201000679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : configmap "kube-rbac-proxy" not found Sep 07 10:12:17.214943 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:17.214787 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 10:12:17.214943 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:17.214833 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.214820557 +0000 UTC m=+63.201100887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : secret "insights-runtime-extractor-tls" not found Sep 07 10:12:17.232221 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:17.232194 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqhd9_ba852866-ea68-4579-8b0a-8f2c843d8023/node-ca/0.log" Sep 07 10:12:18.227376 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:18.227344 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pbd8k_9f1d9389-f077-4289-92f0-19197641f90c/serve-healthcheck-canary/0.log" Sep 07 10:12:18.829727 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:18.829684 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-8c94h_be97b92a-7c7c-4cad-a5be-07bd910af233/kube-storage-version-migrator-operator/0.log" Sep 07 10:12:18.836233 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:18.836209 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qhw7" Sep 07 10:12:21.044760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.044709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:21.045149 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.044865 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:21.045149 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.044931 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls podName:6c6b7ec1-a764-41f0-8541-a7b52c6989a2 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:37.044915265 +0000 UTC m=+79.031195580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wzscw" (UID: "6c6b7ec1-a764-41f0-8541-a7b52c6989a2") : secret "cluster-monitoring-operator-tls" not found Sep 07 10:12:21.145983 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.145946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:21.148575 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.148552 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fdc76fd-992d-420d-a565-e78ab94703c0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sh6ql\" (UID: \"7fdc76fd-992d-420d-a565-e78ab94703c0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:21.246927 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.246897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:21.246927 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.246930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.246962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.246991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.247023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247048 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247060 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247105 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:37.247087161 +0000 UTC m=+79.233367480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : configmap references non-existent config key: service-ca.crt Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247126 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:29.247118639 +0000 UTC m=+71.233398951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : configmap "kube-rbac-proxy" not found Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247124 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247130 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247135 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:29.247130453 +0000 UTC m=+71.233410764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : secret "insights-runtime-extractor-tls" not found Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247202 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs podName:be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:37.247186443 +0000 UTC m=+79.233466769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs") pod "router-default-6bd85f49dd-l2v9q" (UID: "be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0") : secret "router-metrics-certs-default" not found Sep 07 10:12:21.247208 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:21.247216 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert podName:1d477d2b-4f2b-41ee-9800-73c0650da064 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:37.247209891 +0000 UTC m=+79.233490204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-l922z" (UID: "1d477d2b-4f2b-41ee-9800-73c0650da064") : secret "networking-console-plugin-cert" not found Sep 07 10:12:21.430177 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.430075 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9xwvz\"" Sep 07 10:12:21.438233 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.438215 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" Sep 07 10:12:21.706242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.706187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql"] Sep 07 10:12:21.944012 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:21.943977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" event={"ID":"7fdc76fd-992d-420d-a565-e78ab94703c0","Type":"ContainerStarted","Data":"c1613866aebbebfd04756b9db5e43f274e04dfd4dda0c37e5ac1f9c2a3ddee7f"} Sep 07 10:12:23.951884 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:23.951844 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" event={"ID":"7fdc76fd-992d-420d-a565-e78ab94703c0","Type":"ContainerStarted","Data":"5e0f1a389aa880d0e703bc296d1f5f84c2aa3aefe16e321fbfc0177380d67819"} Sep 07 10:12:23.952336 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:23.951920 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" event={"ID":"7fdc76fd-992d-420d-a565-e78ab94703c0","Type":"ContainerStarted","Data":"bd13be00f2e5f98600695345a8f6bf2ba33921acbb1880926ca198a47ed88f73"} Sep 07 10:12:23.996809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:23.996754 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sh6ql" podStartSLOduration=17.311771283 podStartE2EDuration="18.996738589s" totalImagesPullingTime="1.684967306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:23.990320993 +0000 UTC m=+65.976601327" watchObservedRunningTime="2026-09-07 10:12:23.996738589 +0000 UTC m=+65.983018966" Sep 07 10:12:24.274821 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.274774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:12:24.275040 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.274876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:12:24.275040 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.274907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:12:24.277392 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.277364 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78daa9-fda7-4dbc-9465-17e251db07f6-metrics-tls\") pod \"dns-default-bfpfk\" (UID: \"7a78daa9-fda7-4dbc-9465-17e251db07f6\") " pod="openshift-dns/dns-default-bfpfk" Sep 07 10:12:24.277504 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.277442 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"image-registry-78f9bbc69b-rrt8w\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:12:24.277732 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.277715 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 10:12:24.285424 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:24.285249 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 07 10:12:24.285424 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:24.285340 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs podName:05166e4c-db23-4e20-b194-d8adbf6ce954 nodeName:}" failed. No retries permitted until 2026-09-07 10:13:28.28531721 +0000 UTC m=+130.271597522 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs") pod "network-metrics-daemon-lb6k7" (UID: "05166e4c-db23-4e20-b194-d8adbf6ce954") : secret "metrics-daemon-secret" not found Sep 07 10:12:24.375633 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.375597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:12:24.378493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.378473 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 10:12:24.388936 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.388913 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0db793e3-82ec-40b8-ab3e-eddc545c4973-original-pull-secret\") pod \"global-pull-secret-syncer-f4j6j\" (UID: \"0db793e3-82ec-40b8-ab3e-eddc545c4973\") " pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:12:24.466039 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.466011 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hpk4\"" Sep 07 10:12:24.474041 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.474022 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:12:24.476593 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.476570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:12:24.479225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.479209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlznh\" (UniqueName: \"kubernetes.io/projected/94d5e68e-e01b-421b-b544-fe76fd212ed4-kube-api-access-nlznh\") pod \"network-check-target-nsmhj\" (UID: \"94d5e68e-e01b-421b-b544-fe76fd212ed4\") " pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:12:24.481662 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.481642 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8d2\"" Sep 07 10:12:24.490364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.490345 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bfpfk" Sep 07 10:12:24.649785 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.649751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4j6j" Sep 07 10:12:24.659365 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.659340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9xn8\"" Sep 07 10:12:24.667633 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.667600 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:12:24.754260 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.754227 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f9bbc69b-rrt8w"] Sep 07 10:12:24.757388 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:24.757361 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4496a2f9_ed1d_4e70_919d_b4968349702f.slice/crio-438233432f790d1b73a62e3022fbf0b159ebab3626aaee3feb54f9b08a53d752 WatchSource:0}: Error finding container 438233432f790d1b73a62e3022fbf0b159ebab3626aaee3feb54f9b08a53d752: Status 404 returned error can't find the container with id 438233432f790d1b73a62e3022fbf0b159ebab3626aaee3feb54f9b08a53d752 Sep 07 10:12:24.794941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.794904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bfpfk"] Sep 07 10:12:24.796963 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:24.796942 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a78daa9_fda7_4dbc_9465_17e251db07f6.slice/crio-6cf77619a16045c75d780c4a51424ae4b9cd3b2df425d0ef789f205753b7ddc2 WatchSource:0}: Error finding container 6cf77619a16045c75d780c4a51424ae4b9cd3b2df425d0ef789f205753b7ddc2: Status 404 returned error can't find the container with id 6cf77619a16045c75d780c4a51424ae4b9cd3b2df425d0ef789f205753b7ddc2 Sep 07 10:12:24.921582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.921551 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f4j6j"] Sep 07 10:12:24.924800 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:24.924772 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db793e3_82ec_40b8_ab3e_eddc545c4973.slice/crio-d1f9a708e2b2978c55b44b27c56749180f85d9ea7e190ffbe54cf9e0032678d7 WatchSource:0}: Error finding container d1f9a708e2b2978c55b44b27c56749180f85d9ea7e190ffbe54cf9e0032678d7: Status 404 returned error can't find the container with id d1f9a708e2b2978c55b44b27c56749180f85d9ea7e190ffbe54cf9e0032678d7 Sep 07 10:12:24.946193 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.946148 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nsmhj"] Sep 07 10:12:24.948283 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:24.948259 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d5e68e_e01b_421b_b544_fe76fd212ed4.slice/crio-34cf1a7dbc5444d8aeba8bfd9e7c566839fed9eff24c929af9b0b276800cc38d WatchSource:0}: Error finding container 34cf1a7dbc5444d8aeba8bfd9e7c566839fed9eff24c929af9b0b276800cc38d: Status 404 returned error can't find the container with id 34cf1a7dbc5444d8aeba8bfd9e7c566839fed9eff24c929af9b0b276800cc38d Sep 07 10:12:24.956069 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.956042 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nsmhj" event={"ID":"94d5e68e-e01b-421b-b544-fe76fd212ed4","Type":"ContainerStarted","Data":"34cf1a7dbc5444d8aeba8bfd9e7c566839fed9eff24c929af9b0b276800cc38d"} Sep 07 10:12:24.957930 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.957907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" event={"ID":"4496a2f9-ed1d-4e70-919d-b4968349702f","Type":"ContainerStarted","Data":"a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee"} Sep 07 10:12:24.958034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.957951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" event={"ID":"4496a2f9-ed1d-4e70-919d-b4968349702f","Type":"ContainerStarted","Data":"438233432f790d1b73a62e3022fbf0b159ebab3626aaee3feb54f9b08a53d752"} Sep 07 10:12:24.958034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.957979 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:12:24.959103 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.959084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfpfk" event={"ID":"7a78daa9-fda7-4dbc-9465-17e251db07f6","Type":"ContainerStarted","Data":"6cf77619a16045c75d780c4a51424ae4b9cd3b2df425d0ef789f205753b7ddc2"} Sep 07 10:12:24.960285 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.960263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f4j6j" event={"ID":"0db793e3-82ec-40b8-ab3e-eddc545c4973","Type":"ContainerStarted","Data":"d1f9a708e2b2978c55b44b27c56749180f85d9ea7e190ffbe54cf9e0032678d7"} Sep 07 10:12:24.979270 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:24.979197 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" podStartSLOduration=65.979175942 podStartE2EDuration="1m5.979175942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:24.977390375 +0000 UTC m=+66.963670710" watchObservedRunningTime="2026-09-07 10:12:24.979175942 +0000 UTC m=+66.965456268" Sep 07 10:12:25.966694 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:25.966654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nsmhj" event={"ID":"94d5e68e-e01b-421b-b544-fe76fd212ed4","Type":"ContainerStarted","Data":"2e0870481f2d72c9c931d494e2740d2bade19426bc7b19db7c07e27c05d15f0e"} Sep 07 10:12:26.973215 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:26.973183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfpfk" event={"ID":"7a78daa9-fda7-4dbc-9465-17e251db07f6","Type":"ContainerStarted","Data":"8817867676aff6be3e2343112631a249b665b5be90b8e6c1e3cd25f5bda81b4e"} Sep 07 10:12:26.973640 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:26.973247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfpfk" event={"ID":"7a78daa9-fda7-4dbc-9465-17e251db07f6","Type":"ContainerStarted","Data":"df411aabd44f98573db1ac06fa0764b776f27ad7d620f884b0f11aa8b11cc809"} Sep 07 10:12:26.973640 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:26.973366 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:12:26.992992 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:26.992928 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nsmhj" podStartSLOduration=68.992907883 podStartE2EDuration="1m8.992907883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:25.984106385 +0000 UTC m=+67.970386721" watchObservedRunningTime="2026-09-07 10:12:26.992907883 +0000 UTC m=+68.979188219" Sep 07 10:12:26.993217 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:26.993040 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bfpfk" podStartSLOduration=33.091028221 podStartE2EDuration="34.99303281s" totalImagesPullingTime="1.902004589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:26.991133768 +0000 UTC m=+68.977414104" watchObservedRunningTime="2026-09-07 10:12:26.99303281 +0000 UTC m=+68.979313146" Sep 07 10:12:27.977941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:27.977909 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bfpfk" Sep 07 10:12:28.539728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:28.539698 2565 scope.go:117] "RemoveContainer" containerID="4812cb1e5d3e3f9ee4ebf439833608c9a8cd523cdbf6462fc4a364c3acef1952" Sep 07 10:12:29.319511 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:29.319472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:29.319912 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:29.319550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:29.319912 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:29.319623 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 10:12:29.319912 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:29.319695 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls podName:60e57c56-bb1b-42cd-a21c-90a1cae39c71 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:45.319679109 +0000 UTC m=+87.305959438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs2dm" (UID: "60e57c56-bb1b-42cd-a21c-90a1cae39c71") : secret "insights-runtime-extractor-tls" not found Sep 07 10:12:29.320067 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:29.319983 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60e57c56-bb1b-42cd-a21c-90a1cae39c71-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:29.986566 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:29.986464 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f4j6j" event={"ID":"0db793e3-82ec-40b8-ab3e-eddc545c4973","Type":"ContainerStarted","Data":"b734d60797c48b3d122b158428703410ec70a3b6241af40dce44268bb0a13d9e"} Sep 07 10:12:29.988595 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:29.988571 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:12:29.988708 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:29.988688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" event={"ID":"1749d01c-371e-42dc-9c83-d31ecb1b2788","Type":"ContainerStarted","Data":"cc92cddd568271e88b31e474f6731d05fd23f62d7078d675c341db944b3af0ec"} Sep 07 10:12:29.988959 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:29.988942 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:30.001532 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:30.001482 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f4j6j" podStartSLOduration=67.199648173 podStartE2EDuration="1m12.001466565s" totalImagesPullingTime="4.801818392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:30.001279315 +0000 UTC m=+71.987559653" watchObservedRunningTime="2026-09-07 10:12:30.001466565 +0000 UTC m=+71.987746900" Sep 07 10:12:30.016731 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:30.016687 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" podStartSLOduration=19.418933189 podStartE2EDuration="25.016673132s" totalImagesPullingTime="5.597739943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:30.015597092 +0000 UTC m=+72.001877425" watchObservedRunningTime="2026-09-07 10:12:30.016673132 +0000 UTC m=+72.002953468" Sep 07 10:12:30.647796 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:30.647769 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-nbkdc" Sep 07 10:12:37.084724 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.084679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:37.087245 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.087220 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c6b7ec1-a764-41f0-8541-a7b52c6989a2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wzscw\" (UID: \"6c6b7ec1-a764-41f0-8541-a7b52c6989a2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:37.286903 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.286871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:37.286903 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.286909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:37.287128 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.287077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:37.287696 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.287673 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-service-ca-bundle\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:37.289323 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.289296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d477d2b-4f2b-41ee-9800-73c0650da064-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-l922z\" (UID: \"1d477d2b-4f2b-41ee-9800-73c0650da064\") " pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:37.289487 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.289466 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0-metrics-certs\") pod \"router-default-6bd85f49dd-l2v9q\" (UID: \"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0\") " pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:37.301597 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.301580 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xkndp\"" Sep 07 10:12:37.309518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.309501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" Sep 07 10:12:37.451767 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.451739 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7472n\"" Sep 07 10:12:37.459947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.459918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" Sep 07 10:12:37.479426 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.479404 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-g7ptg\"" Sep 07 10:12:37.487548 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.487525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:37.577913 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.577889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw"] Sep 07 10:12:37.580727 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:37.580702 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6b7ec1_a764_41f0_8541_a7b52c6989a2.slice/crio-254d79b384cb1649704de4f00aef4d8743ee86ac5f27c90fd07a89cae431b853 WatchSource:0}: Error finding container 254d79b384cb1649704de4f00aef4d8743ee86ac5f27c90fd07a89cae431b853: Status 404 returned error can't find the container with id 254d79b384cb1649704de4f00aef4d8743ee86ac5f27c90fd07a89cae431b853 Sep 07 10:12:37.725023 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.724996 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-l922z"] Sep 07 10:12:37.727651 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:37.727626 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d477d2b_4f2b_41ee_9800_73c0650da064.slice/crio-f5150ddfc145899eb6eeb3541b8d55e64484c59627fcbefc8e05e682b494de03 WatchSource:0}: Error finding container f5150ddfc145899eb6eeb3541b8d55e64484c59627fcbefc8e05e682b494de03: Status 404 returned error can't find the container with id f5150ddfc145899eb6eeb3541b8d55e64484c59627fcbefc8e05e682b494de03 Sep 07 10:12:37.759793 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.759766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bd85f49dd-l2v9q"] Sep 07 10:12:37.762482 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:37.762462 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1f4ce2_9f6c_4b8e_8d87_2f7250b78ad0.slice/crio-deb17ed424f148b620a167ba1defb4af18ea36d1802071384bde72743ced08f1 WatchSource:0}: Error finding container deb17ed424f148b620a167ba1defb4af18ea36d1802071384bde72743ced08f1: Status 404 returned error can't find the container with id deb17ed424f148b620a167ba1defb4af18ea36d1802071384bde72743ced08f1 Sep 07 10:12:37.982884 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:37.982813 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bfpfk" Sep 07 10:12:38.022298 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.022260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" event={"ID":"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0","Type":"ContainerStarted","Data":"ec6deb6ded3f2b683149b453a64c97a26eee9d066a0ad9ba96c892a6ddc46e8f"} Sep 07 10:12:38.022496 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.022315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" event={"ID":"be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0","Type":"ContainerStarted","Data":"deb17ed424f148b620a167ba1defb4af18ea36d1802071384bde72743ced08f1"} Sep 07 10:12:38.023432 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.023393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" event={"ID":"1d477d2b-4f2b-41ee-9800-73c0650da064","Type":"ContainerStarted","Data":"f5150ddfc145899eb6eeb3541b8d55e64484c59627fcbefc8e05e682b494de03"} Sep 07 10:12:38.024369 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.024353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" event={"ID":"6c6b7ec1-a764-41f0-8541-a7b52c6989a2","Type":"ContainerStarted","Data":"254d79b384cb1649704de4f00aef4d8743ee86ac5f27c90fd07a89cae431b853"} Sep 07 10:12:38.047490 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.047412 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" podStartSLOduration=33.047396826 podStartE2EDuration="33.047396826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:38.047244375 +0000 UTC m=+80.033524708" watchObservedRunningTime="2026-09-07 10:12:38.047396826 +0000 UTC m=+80.033677161" Sep 07 10:12:38.188010 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.187922 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-tcmkq"] Sep 07 10:12:38.191624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.191604 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-tcmkq" Sep 07 10:12:38.194113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.194082 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 07 10:12:38.194234 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.194183 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 07 10:12:38.194345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.194332 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5sk6j\"" Sep 07 10:12:38.203788 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.203766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-tcmkq"] Sep 07 10:12:38.295875 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.295810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnxn\" (UniqueName: \"kubernetes.io/projected/2af8be4d-808e-4d17-b005-70c6f7717297-kube-api-access-ljnxn\") pod \"downloads-85d98c998f-tcmkq\" (UID: \"2af8be4d-808e-4d17-b005-70c6f7717297\") " pod="openshift-console/downloads-85d98c998f-tcmkq" Sep 07 10:12:38.396662 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.396625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnxn\" (UniqueName: \"kubernetes.io/projected/2af8be4d-808e-4d17-b005-70c6f7717297-kube-api-access-ljnxn\") pod \"downloads-85d98c998f-tcmkq\" (UID: \"2af8be4d-808e-4d17-b005-70c6f7717297\") " pod="openshift-console/downloads-85d98c998f-tcmkq" Sep 07 10:12:38.406709 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.406673 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnxn\" (UniqueName: \"kubernetes.io/projected/2af8be4d-808e-4d17-b005-70c6f7717297-kube-api-access-ljnxn\") pod \"downloads-85d98c998f-tcmkq\" (UID: \"2af8be4d-808e-4d17-b005-70c6f7717297\") " pod="openshift-console/downloads-85d98c998f-tcmkq" Sep 07 10:12:38.488298 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.488227 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:38.491610 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.491572 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:38.504640 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.504605 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-tcmkq" Sep 07 10:12:38.793646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:38.793612 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-tcmkq"] Sep 07 10:12:38.973802 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:38.973774 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af8be4d_808e_4d17_b005_70c6f7717297.slice/crio-c51d64986a1c592a91f770180e7c1581618337a86c7327e6119f87d3baede94a WatchSource:0}: Error finding container c51d64986a1c592a91f770180e7c1581618337a86c7327e6119f87d3baede94a: Status 404 returned error can't find the container with id c51d64986a1c592a91f770180e7c1581618337a86c7327e6119f87d3baede94a Sep 07 10:12:39.029987 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:39.029933 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-tcmkq" event={"ID":"2af8be4d-808e-4d17-b005-70c6f7717297","Type":"ContainerStarted","Data":"c51d64986a1c592a91f770180e7c1581618337a86c7327e6119f87d3baede94a"} Sep 07 10:12:39.030363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:39.030305 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:39.031856 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:39.031651 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6bd85f49dd-l2v9q" Sep 07 10:12:40.037322 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:40.037286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" event={"ID":"1d477d2b-4f2b-41ee-9800-73c0650da064","Type":"ContainerStarted","Data":"42957227cee403371fbe03fc70813d0ed8dfba9aced40ce439c247692871ece5"} Sep 07 10:12:40.039146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:40.039119 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" event={"ID":"6c6b7ec1-a764-41f0-8541-a7b52c6989a2","Type":"ContainerStarted","Data":"993b9ac62c7e9c7cf422a0c5c8307539299934ded5e73d26b662a60398c66812"} Sep 07 10:12:40.052642 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:40.052594 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-l922z" podStartSLOduration=33.279622733 podStartE2EDuration="35.052579786s" totalImagesPullingTime="1.772957053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:40.050713687 +0000 UTC m=+82.036994022" watchObservedRunningTime="2026-09-07 10:12:40.052579786 +0000 UTC m=+82.038860120" Sep 07 10:12:40.065958 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:40.065904 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wzscw" podStartSLOduration=33.14182278 podStartE2EDuration="35.065891214s" totalImagesPullingTime="1.924068434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:40.065153743 +0000 UTC m=+82.051434079" watchObservedRunningTime="2026-09-07 10:12:40.065891214 +0000 UTC m=+82.052171549" Sep 07 10:12:44.480514 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:44.480472 2565 patch_prober.go:28] interesting pod/image-registry-78f9bbc69b-rrt8w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 07 10:12:44.481091 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:44.480536 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" podUID="4496a2f9-ed1d-4e70-919d-b4968349702f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:12:45.361047 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:45.361007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:45.363850 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:45.363827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60e57c56-bb1b-42cd-a21c-90a1cae39c71-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs2dm\" (UID: \"60e57c56-bb1b-42cd-a21c-90a1cae39c71\") " pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:45.591715 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:45.591684 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gp5c5\"" Sep 07 10:12:45.599900 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:45.599877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rs2dm" Sep 07 10:12:45.909976 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:45.909904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rs2dm"] Sep 07 10:12:45.913325 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:45.913290 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e57c56_bb1b_42cd_a21c_90a1cae39c71.slice/crio-20f1ccb81724e20c3597497224a589a60f2021b9a8fdf3a23e6b2c58315a32bb WatchSource:0}: Error finding container 20f1ccb81724e20c3597497224a589a60f2021b9a8fdf3a23e6b2c58315a32bb: Status 404 returned error can't find the container with id 20f1ccb81724e20c3597497224a589a60f2021b9a8fdf3a23e6b2c58315a32bb Sep 07 10:12:45.971555 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:45.971531 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:12:46.064598 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:46.064562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rs2dm" event={"ID":"60e57c56-bb1b-42cd-a21c-90a1cae39c71","Type":"ContainerStarted","Data":"dd224db1c45e765dfcfd9fd3b7cf4b48feb90b9751ceefff555afda6f21aca7c"} Sep 07 10:12:46.064752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:46.064636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rs2dm" event={"ID":"60e57c56-bb1b-42cd-a21c-90a1cae39c71","Type":"ContainerStarted","Data":"20f1ccb81724e20c3597497224a589a60f2021b9a8fdf3a23e6b2c58315a32bb"} Sep 07 10:12:47.070865 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:47.070832 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rs2dm" event={"ID":"60e57c56-bb1b-42cd-a21c-90a1cae39c71","Type":"ContainerStarted","Data":"457e92bacf3c53ca8d3cdf4ee4a7eb4e5254571956a96d8747db2d35b0e4e796"} Sep 07 10:12:56.111260 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.111220 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-tcmkq" event={"ID":"2af8be4d-808e-4d17-b005-70c6f7717297","Type":"ContainerStarted","Data":"6ac5dccf4a4f3a15933836a3608a4edf932848444acca55fdf62c84c35444283"} Sep 07 10:12:56.112831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.112803 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-tcmkq" Sep 07 10:12:56.115677 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.115650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rs2dm" event={"ID":"60e57c56-bb1b-42cd-a21c-90a1cae39c71","Type":"ContainerStarted","Data":"ae82a984de9799204b0f926934610e9e9182dae8d0292d75476897a04dfbb2a7"} Sep 07 10:12:56.126618 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.126593 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-tcmkq" Sep 07 10:12:56.139763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.139708 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-tcmkq" podStartSLOduration=1.663451663 podStartE2EDuration="18.139689809s" totalImagesPullingTime="16.476238146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:56.138059804 +0000 UTC m=+98.124340139" watchObservedRunningTime="2026-09-07 10:12:56.139689809 +0000 UTC m=+98.125970144" Sep 07 10:12:56.189784 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.189715 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rs2dm" podStartSLOduration=33.906529236 podStartE2EDuration="43.189694185s" totalImagesPullingTime="9.283164949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:12:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:56.188119039 +0000 UTC m=+98.174399386" watchObservedRunningTime="2026-09-07 10:12:56.189694185 +0000 UTC m=+98.175974520" Sep 07 10:12:56.493297 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.493263 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r"] Sep 07 10:12:56.499481 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.499450 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xnqbp"] Sep 07 10:12:56.499680 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.499591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.502115 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.502088 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 07 10:12:56.503457 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.503433 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 07 10:12:56.503574 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.503536 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qcz9g\"" Sep 07 10:12:56.503670 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.503649 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 07 10:12:56.503780 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.503760 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 07 10:12:56.504148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.504126 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.506521 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.506501 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 07 10:12:56.506688 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.506669 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 07 10:12:56.506762 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.506546 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 07 10:12:56.506762 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.506605 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s5st7\"" Sep 07 10:12:56.509398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.509376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r"] Sep 07 10:12:56.562578 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.562578 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.562825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-wtmp\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.562825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9674401-ea1d-4172-a89c-59f5089e6531-metrics-client-ca\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.562825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562737 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-tls\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.562825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.562825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-root\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.563055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.563055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-textfile\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.563055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3417365e-4283-4549-8f2f-b0c68d94931f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.563055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-accelerators-collector-config\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.563055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.562991 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frs9r\" (UniqueName: \"kubernetes.io/projected/a9674401-ea1d-4172-a89c-59f5089e6531-kube-api-access-frs9r\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.563055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.563037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-sys\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.563358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.563065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3417365e-4283-4549-8f2f-b0c68d94931f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.563358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.563089 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fmn\" (UniqueName: \"kubernetes.io/projected/3417365e-4283-4549-8f2f-b0c68d94931f-kube-api-access-79fmn\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.663998 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.663957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frs9r\" (UniqueName: \"kubernetes.io/projected/a9674401-ea1d-4172-a89c-59f5089e6531-kube-api-access-frs9r\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-sys\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3417365e-4283-4549-8f2f-b0c68d94931f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.664219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fmn\" (UniqueName: \"kubernetes.io/projected/3417365e-4283-4549-8f2f-b0c68d94931f-kube-api-access-79fmn\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.664427 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.664427 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664427 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-wtmp\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664427 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9674401-ea1d-4172-a89c-59f5089e6531-metrics-client-ca\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664427 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-tls\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.664674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-root\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.664674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-textfile\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3417365e-4283-4549-8f2f-b0c68d94931f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.664674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-accelerators-collector-config\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.664961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.664945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-root\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.665386 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.665356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-accelerators-collector-config\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.665541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.665517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9674401-ea1d-4172-a89c-59f5089e6531-metrics-client-ca\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.666053 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:56.665567 2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 07 10:12:56.666257 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:12:56.666243 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-tls podName:3417365e-4283-4549-8f2f-b0c68d94931f nodeName:}" failed. No retries permitted until 2026-09-07 10:12:57.166220608 +0000 UTC m=+99.152500920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-mvt4r" (UID: "3417365e-4283-4549-8f2f-b0c68d94931f") : secret "kube-state-metrics-tls" not found Sep 07 10:12:56.666354 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.665682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-sys\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.666622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.665758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-textfile\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.666622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.665805 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.666781 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.666053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3417365e-4283-4549-8f2f-b0c68d94931f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.666781 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.666286 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3417365e-4283-4549-8f2f-b0c68d94931f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.666781 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.665702 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-wtmp\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.668544 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.668517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.668679 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.668555 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.668911 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.668891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9674401-ea1d-4172-a89c-59f5089e6531-node-exporter-tls\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.675120 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.675096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frs9r\" (UniqueName: \"kubernetes.io/projected/a9674401-ea1d-4172-a89c-59f5089e6531-kube-api-access-frs9r\") pod \"node-exporter-xnqbp\" (UID: \"a9674401-ea1d-4172-a89c-59f5089e6531\") " pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.680796 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.680767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fmn\" (UniqueName: \"kubernetes.io/projected/3417365e-4283-4549-8f2f-b0c68d94931f-kube-api-access-79fmn\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:56.822434 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:56.822362 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xnqbp" Sep 07 10:12:56.854143 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:12:56.853544 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9674401_ea1d_4172_a89c_59f5089e6531.slice/crio-ee2b1be5e852ef5731153ec793bbeed9853c00e653393066f6fe7d8ba341a0d0 WatchSource:0}: Error finding container ee2b1be5e852ef5731153ec793bbeed9853c00e653393066f6fe7d8ba341a0d0: Status 404 returned error can't find the container with id ee2b1be5e852ef5731153ec793bbeed9853c00e653393066f6fe7d8ba341a0d0 Sep 07 10:12:57.122230 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:57.122119 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xnqbp" event={"ID":"a9674401-ea1d-4172-a89c-59f5089e6531","Type":"ContainerStarted","Data":"ee2b1be5e852ef5731153ec793bbeed9853c00e653393066f6fe7d8ba341a0d0"} Sep 07 10:12:57.170101 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:57.170064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:57.173075 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:57.173052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3417365e-4283-4549-8f2f-b0c68d94931f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mvt4r\" (UID: \"3417365e-4283-4549-8f2f-b0c68d94931f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:57.415679 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:57.415400 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" Sep 07 10:12:57.743329 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:57.743292 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r"] Sep 07 10:12:57.980322 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:57.980292 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nsmhj" Sep 07 10:12:58.129813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:58.129699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" event={"ID":"3417365e-4283-4549-8f2f-b0c68d94931f","Type":"ContainerStarted","Data":"39e7ece9ee059752cb3f03b31d93e6b1555322bbb21e49febaaf6e7ae932e896"} Sep 07 10:12:59.135664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:59.135627 2565 generic.go:356] "Generic (PLEG): container finished" podID="a9674401-ea1d-4172-a89c-59f5089e6531" containerID="d284564f687c2355d5c74f56d799e5f686d47fed555186284a36bba31f9ad1c8" exitCode=0 Sep 07 10:12:59.136132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:12:59.135721 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xnqbp" event={"ID":"a9674401-ea1d-4172-a89c-59f5089e6531","Type":"ContainerDied","Data":"d284564f687c2355d5c74f56d799e5f686d47fed555186284a36bba31f9ad1c8"} Sep 07 10:13:00.144343 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:00.144293 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xnqbp" event={"ID":"a9674401-ea1d-4172-a89c-59f5089e6531","Type":"ContainerStarted","Data":"0277d693ac3f4372da2084d6348ec6094dd47e9e0ec20910166651430535bdd4"} Sep 07 10:13:00.144801 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:00.144381 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xnqbp" event={"ID":"a9674401-ea1d-4172-a89c-59f5089e6531","Type":"ContainerStarted","Data":"e0c2e0ca06ed0c1ad189155b14ca14959e28fdabdbbd1b93ce55261898a42e5f"} Sep 07 10:13:00.148511 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:00.148422 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" event={"ID":"3417365e-4283-4549-8f2f-b0c68d94931f","Type":"ContainerStarted","Data":"5566d1d380b0a0057791aca867993cd6d02264b93a1bf0a60f1f3a1cd9f7e79d"} Sep 07 10:13:00.364187 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:00.360730 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xnqbp" podStartSLOduration=2.449655784 podStartE2EDuration="4.3607031s" totalImagesPullingTime="1.817662058s" totalInitContainerRuntime="93.385258ms" isStatefulPod=true podCreationTimestamp="2026-09-07 10:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:00.194537721 +0000 UTC m=+102.180818056" watchObservedRunningTime="2026-09-07 10:13:00.3607031 +0000 UTC m=+102.346983436" Sep 07 10:13:00.364187 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:00.361844 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78f9bbc69b-rrt8w"] Sep 07 10:13:01.156773 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.156737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" event={"ID":"3417365e-4283-4549-8f2f-b0c68d94931f","Type":"ContainerStarted","Data":"9d6e0535a6c91f919b5ce7ad42a3abc760762e8aa5e7ccbcc970978312c120e8"} Sep 07 10:13:01.157208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.156799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" event={"ID":"3417365e-4283-4549-8f2f-b0c68d94931f","Type":"ContainerStarted","Data":"1d8b786267e887e038f7ba001b4f50456938987ad33e5dedb56aa1a32a9536fd"} Sep 07 10:13:01.179961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.179908 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mvt4r" podStartSLOduration=2.960659199 podStartE2EDuration="5.179889213s" totalImagesPullingTime="2.219230014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:01.179558328 +0000 UTC m=+103.165838665" watchObservedRunningTime="2026-09-07 10:13:01.179889213 +0000 UTC m=+103.166169548" Sep 07 10:13:01.234851 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.234819 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc"] Sep 07 10:13:01.262429 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.262390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc"] Sep 07 10:13:01.262617 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.262516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:01.265281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.265261 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gmf65\"" Sep 07 10:13:01.265402 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.265300 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 07 10:13:01.313490 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.313449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/46d5d069-14bb-4fbc-be12-2c85d05bd81b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fnvkc\" (UID: \"46d5d069-14bb-4fbc-be12-2c85d05bd81b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:01.313663 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.313576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7jr\" (UniqueName: \"kubernetes.io/projected/46d5d069-14bb-4fbc-be12-2c85d05bd81b-kube-api-access-7c7jr\") pod \"monitoring-plugin-85bbbbbf4-fnvkc\" (UID: \"46d5d069-14bb-4fbc-be12-2c85d05bd81b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:01.415142 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.415062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/46d5d069-14bb-4fbc-be12-2c85d05bd81b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fnvkc\" (UID: \"46d5d069-14bb-4fbc-be12-2c85d05bd81b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:01.415142 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.415126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7jr\" (UniqueName: \"kubernetes.io/projected/46d5d069-14bb-4fbc-be12-2c85d05bd81b-kube-api-access-7c7jr\") pod \"monitoring-plugin-85bbbbbf4-fnvkc\" (UID: \"46d5d069-14bb-4fbc-be12-2c85d05bd81b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:01.417863 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.417836 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/46d5d069-14bb-4fbc-be12-2c85d05bd81b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fnvkc\" (UID: \"46d5d069-14bb-4fbc-be12-2c85d05bd81b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:01.426455 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.426434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7jr\" (UniqueName: \"kubernetes.io/projected/46d5d069-14bb-4fbc-be12-2c85d05bd81b-kube-api-access-7c7jr\") pod \"monitoring-plugin-85bbbbbf4-fnvkc\" (UID: \"46d5d069-14bb-4fbc-be12-2c85d05bd81b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:01.573588 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.573555 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:01.670249 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.667982 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5db95d656f-w9vx6"] Sep 07 10:13:01.694626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.694595 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5db95d656f-w9vx6"] Sep 07 10:13:01.694800 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.694708 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.700008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.698910 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 07 10:13:01.700008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.699079 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 07 10:13:01.700008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.699111 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 07 10:13:01.700008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.699250 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-h6ddq\"" Sep 07 10:13:01.700008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.699347 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 07 10:13:01.700008 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.699375 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 07 10:13:01.706081 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.705950 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 07 10:13:01.818176 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.818134 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.818387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.818220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5tz\" (UniqueName: \"kubernetes.io/projected/3aa69b0b-c80d-4170-899c-f81489f70610-kube-api-access-zf5tz\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.818387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.818252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-serving-certs-ca-bundle\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.818387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.818276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-secret-telemeter-client\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.818387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.818306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-telemeter-client-tls\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.818387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.818378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-federate-client-tls\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.818576 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.818416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.818576 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.818472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-metrics-client-ca\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.866606 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.866561 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc"] Sep 07 10:13:01.871001 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:13:01.870973 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d5d069_14bb_4fbc_be12_2c85d05bd81b.slice/crio-7431833410c8cb99188260588066a0564edd4f6a5d8588cf903a50a458cf09de WatchSource:0}: Error finding container 7431833410c8cb99188260588066a0564edd4f6a5d8588cf903a50a458cf09de: Status 404 returned error can't find the container with id 7431833410c8cb99188260588066a0564edd4f6a5d8588cf903a50a458cf09de Sep 07 10:13:01.919527 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.919480 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-metrics-client-ca\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.919714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.919568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.919714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.919610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5tz\" (UniqueName: \"kubernetes.io/projected/3aa69b0b-c80d-4170-899c-f81489f70610-kube-api-access-zf5tz\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.919714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.919655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-serving-certs-ca-bundle\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.919714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.919689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-secret-telemeter-client\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.919714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.919714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-telemeter-client-tls\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.919985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.919734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-federate-client-tls\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.919985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.919763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.920242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.920194 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-metrics-client-ca\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.920958 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.920877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-serving-certs-ca-bundle\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.921059 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.921001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aa69b0b-c80d-4170-899c-f81489f70610-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.922961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.922720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.922961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.922927 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-secret-telemeter-client\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.923097 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.923001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-telemeter-client-tls\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.923140 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.923084 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3aa69b0b-c80d-4170-899c-f81489f70610-federate-client-tls\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:01.928107 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:01.928077 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5tz\" (UniqueName: \"kubernetes.io/projected/3aa69b0b-c80d-4170-899c-f81489f70610-kube-api-access-zf5tz\") pod \"telemeter-client-5db95d656f-w9vx6\" (UID: \"3aa69b0b-c80d-4170-899c-f81489f70610\") " pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:02.009777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.009738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" Sep 07 10:13:02.162829 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.162760 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" event={"ID":"46d5d069-14bb-4fbc-be12-2c85d05bd81b","Type":"ContainerStarted","Data":"7431833410c8cb99188260588066a0564edd4f6a5d8588cf903a50a458cf09de"} Sep 07 10:13:02.326625 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.326542 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5db95d656f-w9vx6"] Sep 07 10:13:02.331205 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:13:02.331174 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa69b0b_c80d_4170_899c_f81489f70610.slice/crio-f790a2716af3b4ee045d5bb1130d464043a511ca3f6773a567074c02625f0f34 WatchSource:0}: Error finding container f790a2716af3b4ee045d5bb1130d464043a511ca3f6773a567074c02625f0f34: Status 404 returned error can't find the container with id f790a2716af3b4ee045d5bb1130d464043a511ca3f6773a567074c02625f0f34 Sep 07 10:13:02.345327 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.345297 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c76479bc7-w98th"] Sep 07 10:13:02.366533 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.366494 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c76479bc7-w98th"] Sep 07 10:13:02.366776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.366630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.369371 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.369350 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ndkq9\"" Sep 07 10:13:02.370112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.369631 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 07 10:13:02.370112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.369683 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 07 10:13:02.370112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.369820 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 07 10:13:02.370112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.369941 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 07 10:13:02.370112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.370000 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 07 10:13:02.374971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.374951 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 07 10:13:02.425837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.425803 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-oauth-config\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.425999 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.425902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-serving-cert\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.426068 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.425989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-service-ca\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.426068 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.426025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl47p\" (UniqueName: \"kubernetes.io/projected/bde39ddf-db09-473d-acfd-31dd8e2ef9be-kube-api-access-zl47p\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.426219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.426095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-trusted-ca-bundle\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.426219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.426129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-config\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.426219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.426152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-oauth-serving-cert\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.527665 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.527634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-oauth-config\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.527899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.527715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-serving-cert\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.527899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.527763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-service-ca\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.527899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.527787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl47p\" (UniqueName: \"kubernetes.io/projected/bde39ddf-db09-473d-acfd-31dd8e2ef9be-kube-api-access-zl47p\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.527899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.527832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-trusted-ca-bundle\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.527899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.527861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-config\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.527899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.527892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-oauth-serving-cert\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.529066 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.529039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-config\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.529342 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.529316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-trusted-ca-bundle\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.529642 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.529542 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-oauth-serving-cert\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.529871 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.529836 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-service-ca\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.531445 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.531422 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-oauth-config\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.532270 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.532225 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-serving-cert\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.537021 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.536993 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl47p\" (UniqueName: \"kubernetes.io/projected/bde39ddf-db09-473d-acfd-31dd8e2ef9be-kube-api-access-zl47p\") pod \"console-6c76479bc7-w98th\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.680081 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.679930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:02.987473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:02.987443 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c76479bc7-w98th"] Sep 07 10:13:02.991437 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:13:02.991350 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde39ddf_db09_473d_acfd_31dd8e2ef9be.slice/crio-6fb1ba401c464be2506a3756406308ef1662dc950f0f2e6fe4a2c4276cc177c8 WatchSource:0}: Error finding container 6fb1ba401c464be2506a3756406308ef1662dc950f0f2e6fe4a2c4276cc177c8: Status 404 returned error can't find the container with id 6fb1ba401c464be2506a3756406308ef1662dc950f0f2e6fe4a2c4276cc177c8 Sep 07 10:13:03.178407 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:03.178313 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c76479bc7-w98th" event={"ID":"bde39ddf-db09-473d-acfd-31dd8e2ef9be","Type":"ContainerStarted","Data":"6fb1ba401c464be2506a3756406308ef1662dc950f0f2e6fe4a2c4276cc177c8"} Sep 07 10:13:03.180434 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:03.180200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" event={"ID":"3aa69b0b-c80d-4170-899c-f81489f70610","Type":"ContainerStarted","Data":"f790a2716af3b4ee045d5bb1130d464043a511ca3f6773a567074c02625f0f34"} Sep 07 10:13:06.194487 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:06.194441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" event={"ID":"46d5d069-14bb-4fbc-be12-2c85d05bd81b","Type":"ContainerStarted","Data":"fe6b944d4ff87c07d314188db812ca40cac4f12c83fa43e86118663ec893f933"} Sep 07 10:13:06.194997 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:06.194632 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:06.196517 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:06.196484 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" event={"ID":"3aa69b0b-c80d-4170-899c-f81489f70610","Type":"ContainerStarted","Data":"59acbc5cc66db0077c8d7045c3c184f8cd68107e57b48d1e0445f002d41915da"} Sep 07 10:13:06.200938 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:06.200912 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" Sep 07 10:13:06.213174 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:06.213110 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fnvkc" podStartSLOduration=2.002571312 podStartE2EDuration="5.213096551s" totalImagesPullingTime="3.210525239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:06.210959512 +0000 UTC m=+108.197239847" watchObservedRunningTime="2026-09-07 10:13:06.213096551 +0000 UTC m=+108.199376909" Sep 07 10:13:08.205600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:08.205563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c76479bc7-w98th" event={"ID":"bde39ddf-db09-473d-acfd-31dd8e2ef9be","Type":"ContainerStarted","Data":"6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a"} Sep 07 10:13:08.223080 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:08.223029 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c76479bc7-w98th" podStartSLOduration=1.844224893 podStartE2EDuration="6.223012012s" totalImagesPullingTime="4.378787119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:08.222033094 +0000 UTC m=+110.208313428" watchObservedRunningTime="2026-09-07 10:13:08.223012012 +0000 UTC m=+110.209292345" Sep 07 10:13:09.211097 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:09.211056 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" event={"ID":"3aa69b0b-c80d-4170-899c-f81489f70610","Type":"ContainerStarted","Data":"3c933484965c0a32f6690c1e2f521f1719b80f9216ba36038c7b3cd29c2aa06f"} Sep 07 10:13:09.211501 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:09.211149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" event={"ID":"3aa69b0b-c80d-4170-899c-f81489f70610","Type":"ContainerStarted","Data":"054b90bf1046548d6899e9b47564108d74d3ecd0f9d7dfab9e931051e3be66a8"} Sep 07 10:13:09.240044 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:09.239992 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5db95d656f-w9vx6" podStartSLOduration=2.46492616 podStartE2EDuration="8.239974019s" totalImagesPullingTime="5.775047859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:13:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:09.238149948 +0000 UTC m=+111.224430283" watchObservedRunningTime="2026-09-07 10:13:09.239974019 +0000 UTC m=+111.226254353" Sep 07 10:13:09.328986 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:09.328956 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c76479bc7-w98th"] Sep 07 10:13:12.680824 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:12.680780 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:25.393313 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.393271 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" podUID="4496a2f9-ed1d-4e70-919d-b4968349702f" containerName="registry" containerID="cri-o://a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee" gracePeriod=30 Sep 07 10:13:25.660733 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.660675 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:13:25.740966 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.740936 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4496a2f9-ed1d-4e70-919d-b4968349702f-ca-trust-extracted\") pod \"4496a2f9-ed1d-4e70-919d-b4968349702f\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " Sep 07 10:13:25.740966 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.740969 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-trusted-ca\") pod \"4496a2f9-ed1d-4e70-919d-b4968349702f\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " Sep 07 10:13:25.741231 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.740998 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-installation-pull-secrets\") pod \"4496a2f9-ed1d-4e70-919d-b4968349702f\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " Sep 07 10:13:25.741231 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.741208 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") pod \"4496a2f9-ed1d-4e70-919d-b4968349702f\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " Sep 07 10:13:25.741333 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.741257 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkjp8\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-kube-api-access-kkjp8\") pod \"4496a2f9-ed1d-4e70-919d-b4968349702f\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " Sep 07 10:13:25.741333 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.741321 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-certificates\") pod \"4496a2f9-ed1d-4e70-919d-b4968349702f\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " Sep 07 10:13:25.741440 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.741356 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-image-registry-private-configuration\") pod \"4496a2f9-ed1d-4e70-919d-b4968349702f\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " Sep 07 10:13:25.741440 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.741353 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4496a2f9-ed1d-4e70-919d-b4968349702f" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:13:25.741440 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.741384 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-bound-sa-token\") pod \"4496a2f9-ed1d-4e70-919d-b4968349702f\" (UID: \"4496a2f9-ed1d-4e70-919d-b4968349702f\") " Sep 07 10:13:25.741728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.741675 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-trusted-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:25.741728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.741681 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4496a2f9-ed1d-4e70-919d-b4968349702f" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:13:25.743631 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.743608 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4496a2f9-ed1d-4e70-919d-b4968349702f" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:13:25.743752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.743720 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4496a2f9-ed1d-4e70-919d-b4968349702f" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:13:25.744149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.744120 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4496a2f9-ed1d-4e70-919d-b4968349702f" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:13:25.744281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.744193 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-kube-api-access-kkjp8" (OuterVolumeSpecName: "kube-api-access-kkjp8") pod "4496a2f9-ed1d-4e70-919d-b4968349702f" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f"). InnerVolumeSpecName "kube-api-access-kkjp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:13:25.744281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.744239 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4496a2f9-ed1d-4e70-919d-b4968349702f" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:13:25.750296 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.750264 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4496a2f9-ed1d-4e70-919d-b4968349702f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4496a2f9-ed1d-4e70-919d-b4968349702f" (UID: "4496a2f9-ed1d-4e70-919d-b4968349702f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:13:25.842450 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.842417 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-certificates\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:25.842450 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.842448 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-image-registry-private-configuration\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:25.842450 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.842460 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-bound-sa-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:25.842691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.842468 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4496a2f9-ed1d-4e70-919d-b4968349702f-ca-trust-extracted\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:25.842691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.842477 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4496a2f9-ed1d-4e70-919d-b4968349702f-installation-pull-secrets\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:25.842691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.842485 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-registry-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:25.842691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:25.842493 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjp8\" (UniqueName: \"kubernetes.io/projected/4496a2f9-ed1d-4e70-919d-b4968349702f-kube-api-access-kkjp8\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:26.276974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.276936 2565 generic.go:356] "Generic (PLEG): container finished" podID="4496a2f9-ed1d-4e70-919d-b4968349702f" containerID="a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee" exitCode=0 Sep 07 10:13:26.277148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.276980 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" Sep 07 10:13:26.277148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.277024 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" event={"ID":"4496a2f9-ed1d-4e70-919d-b4968349702f","Type":"ContainerDied","Data":"a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee"} Sep 07 10:13:26.277148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.277078 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f9bbc69b-rrt8w" event={"ID":"4496a2f9-ed1d-4e70-919d-b4968349702f","Type":"ContainerDied","Data":"438233432f790d1b73a62e3022fbf0b159ebab3626aaee3feb54f9b08a53d752"} Sep 07 10:13:26.277148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.277099 2565 scope.go:117] "RemoveContainer" containerID="a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee" Sep 07 10:13:26.291735 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.291717 2565 scope.go:117] "RemoveContainer" containerID="a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee" Sep 07 10:13:26.292037 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:13:26.292016 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee\": container with ID starting with a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee not found: ID does not exist" containerID="a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee" Sep 07 10:13:26.292077 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.292047 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee"} err="failed to get container status \"a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee\": rpc error: code = NotFound desc = could not find container \"a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee\": container with ID starting with a68997f4bd6d1317273431b65d30ad99afa2bf5a2d3e0e454e2301bbc7592bee not found: ID does not exist" Sep 07 10:13:26.297756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.297733 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78f9bbc69b-rrt8w"] Sep 07 10:13:26.301576 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.301556 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78f9bbc69b-rrt8w"] Sep 07 10:13:26.542586 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:26.542506 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4496a2f9-ed1d-4e70-919d-b4968349702f" path="/var/lib/kubelet/pods/4496a2f9-ed1d-4e70-919d-b4968349702f/volumes" Sep 07 10:13:27.282404 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:27.282371 2565 generic.go:356] "Generic (PLEG): container finished" podID="be97b92a-7c7c-4cad-a5be-07bd910af233" containerID="66e3c3f1b36d9dee21e9ffea9b941910919bbfc8a754c757d5e26a09224d850d" exitCode=0 Sep 07 10:13:27.282564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:27.282450 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" event={"ID":"be97b92a-7c7c-4cad-a5be-07bd910af233","Type":"ContainerDied","Data":"66e3c3f1b36d9dee21e9ffea9b941910919bbfc8a754c757d5e26a09224d850d"} Sep 07 10:13:27.282816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:27.282797 2565 scope.go:117] "RemoveContainer" containerID="66e3c3f1b36d9dee21e9ffea9b941910919bbfc8a754c757d5e26a09224d850d" Sep 07 10:13:27.284096 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:27.284076 2565 generic.go:356] "Generic (PLEG): container finished" podID="778d7208-c6a5-45c3-84d5-618bb54a7924" containerID="8a198b61fd05eab08c3daae3601881aab8a3bd055fc8b752eece034ccdf579ff" exitCode=0 Sep 07 10:13:27.284215 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:27.284147 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" event={"ID":"778d7208-c6a5-45c3-84d5-618bb54a7924","Type":"ContainerDied","Data":"8a198b61fd05eab08c3daae3601881aab8a3bd055fc8b752eece034ccdf579ff"} Sep 07 10:13:27.284444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:27.284425 2565 scope.go:117] "RemoveContainer" containerID="8a198b61fd05eab08c3daae3601881aab8a3bd055fc8b752eece034ccdf579ff" Sep 07 10:13:28.290503 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:28.290470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qkcbp" event={"ID":"778d7208-c6a5-45c3-84d5-618bb54a7924","Type":"ContainerStarted","Data":"e641a5ac8c881077c2965ce183cde9e54be4c75a2772fd021e31bfa0cc75b597"} Sep 07 10:13:28.292249 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:28.292225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8c94h" event={"ID":"be97b92a-7c7c-4cad-a5be-07bd910af233","Type":"ContainerStarted","Data":"cffd9ccb279447561f4f1eb9d5b93268341b6fe71610f15a806ec7cf77d7f737"} Sep 07 10:13:28.362492 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:28.362456 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:13:28.365100 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:28.365070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05166e4c-db23-4e20-b194-d8adbf6ce954-metrics-certs\") pod \"network-metrics-daemon-lb6k7\" (UID: \"05166e4c-db23-4e20-b194-d8adbf6ce954\") " pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:13:28.563741 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:28.563668 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cp5xp\"" Sep 07 10:13:28.572064 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:28.572046 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lb6k7" Sep 07 10:13:28.863143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:28.863116 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lb6k7"] Sep 07 10:13:28.865422 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:13:28.865389 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05166e4c_db23_4e20_b194_d8adbf6ce954.slice/crio-a39a35a6564b5091bb953a9a8ba641ad0e147dff079153de45fc1f936aa4a947 WatchSource:0}: Error finding container a39a35a6564b5091bb953a9a8ba641ad0e147dff079153de45fc1f936aa4a947: Status 404 returned error can't find the container with id a39a35a6564b5091bb953a9a8ba641ad0e147dff079153de45fc1f936aa4a947 Sep 07 10:13:29.296848 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:29.296812 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lb6k7" event={"ID":"05166e4c-db23-4e20-b194-d8adbf6ce954","Type":"ContainerStarted","Data":"a39a35a6564b5091bb953a9a8ba641ad0e147dff079153de45fc1f936aa4a947"} Sep 07 10:13:31.306239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:31.306202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lb6k7" event={"ID":"05166e4c-db23-4e20-b194-d8adbf6ce954","Type":"ContainerStarted","Data":"6c799456df07557d48cccdc8e7ac70f4d6954d4cf6758816a42c564a7e7dd869"} Sep 07 10:13:31.306726 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:31.306269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lb6k7" event={"ID":"05166e4c-db23-4e20-b194-d8adbf6ce954","Type":"ContainerStarted","Data":"2ef675937b1fa273dbfba38a37b5fdea0e010cec6c917d5f5c07fc020e0333aa"} Sep 07 10:13:31.322353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:31.322301 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lb6k7" podStartSLOduration=131.705777472 podStartE2EDuration="2m13.322285509s" totalImagesPullingTime="1.616508037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:31.321614297 +0000 UTC m=+133.307894631" watchObservedRunningTime="2026-09-07 10:13:31.322285509 +0000 UTC m=+133.308565843" Sep 07 10:13:35.234742 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.234680 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c76479bc7-w98th" podUID="bde39ddf-db09-473d-acfd-31dd8e2ef9be" containerName="console" containerID="cri-o://6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a" gracePeriod=15 Sep 07 10:13:35.509508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.509450 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:35.629233 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629202 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-serving-cert\") pod \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " Sep 07 10:13:35.629403 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629256 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl47p\" (UniqueName: \"kubernetes.io/projected/bde39ddf-db09-473d-acfd-31dd8e2ef9be-kube-api-access-zl47p\") pod \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " Sep 07 10:13:35.629403 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629283 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-oauth-serving-cert\") pod \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " Sep 07 10:13:35.629403 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629304 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-service-ca\") pod \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " Sep 07 10:13:35.629403 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629347 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-oauth-config\") pod \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " Sep 07 10:13:35.629664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629418 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-config\") pod \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " Sep 07 10:13:35.629664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629444 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-trusted-ca-bundle\") pod \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\" (UID: \"bde39ddf-db09-473d-acfd-31dd8e2ef9be\") " Sep 07 10:13:35.629876 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629832 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-service-ca" (OuterVolumeSpecName: "service-ca") pod "bde39ddf-db09-473d-acfd-31dd8e2ef9be" (UID: "bde39ddf-db09-473d-acfd-31dd8e2ef9be"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:13:35.630207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629866 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bde39ddf-db09-473d-acfd-31dd8e2ef9be" (UID: "bde39ddf-db09-473d-acfd-31dd8e2ef9be"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:13:35.630207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629890 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-config" (OuterVolumeSpecName: "console-config") pod "bde39ddf-db09-473d-acfd-31dd8e2ef9be" (UID: "bde39ddf-db09-473d-acfd-31dd8e2ef9be"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:13:35.630207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.629985 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bde39ddf-db09-473d-acfd-31dd8e2ef9be" (UID: "bde39ddf-db09-473d-acfd-31dd8e2ef9be"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:13:35.631662 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.631632 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bde39ddf-db09-473d-acfd-31dd8e2ef9be" (UID: "bde39ddf-db09-473d-acfd-31dd8e2ef9be"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:13:35.631753 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.631685 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bde39ddf-db09-473d-acfd-31dd8e2ef9be" (UID: "bde39ddf-db09-473d-acfd-31dd8e2ef9be"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:13:35.631753 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.631732 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde39ddf-db09-473d-acfd-31dd8e2ef9be-kube-api-access-zl47p" (OuterVolumeSpecName: "kube-api-access-zl47p") pod "bde39ddf-db09-473d-acfd-31dd8e2ef9be" (UID: "bde39ddf-db09-473d-acfd-31dd8e2ef9be"). InnerVolumeSpecName "kube-api-access-zl47p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:13:35.730831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.730806 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-oauth-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:35.730831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.730828 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:35.730968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.730837 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-trusted-ca-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:35.730968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.730845 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde39ddf-db09-473d-acfd-31dd8e2ef9be-console-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:35.730968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.730853 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl47p\" (UniqueName: \"kubernetes.io/projected/bde39ddf-db09-473d-acfd-31dd8e2ef9be-kube-api-access-zl47p\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:35.730968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.730862 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-oauth-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:35.730968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:35.730870 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde39ddf-db09-473d-acfd-31dd8e2ef9be-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:13:36.326962 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.326925 2565 generic.go:356] "Generic (PLEG): container finished" podID="bde39ddf-db09-473d-acfd-31dd8e2ef9be" containerID="6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a" exitCode=0 Sep 07 10:13:36.327389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.326971 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c76479bc7-w98th" Sep 07 10:13:36.327389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.327003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c76479bc7-w98th" event={"ID":"bde39ddf-db09-473d-acfd-31dd8e2ef9be","Type":"ContainerDied","Data":"6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a"} Sep 07 10:13:36.327389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.327040 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c76479bc7-w98th" event={"ID":"bde39ddf-db09-473d-acfd-31dd8e2ef9be","Type":"ContainerDied","Data":"6fb1ba401c464be2506a3756406308ef1662dc950f0f2e6fe4a2c4276cc177c8"} Sep 07 10:13:36.327389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.327059 2565 scope.go:117] "RemoveContainer" containerID="6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a" Sep 07 10:13:36.335683 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.335667 2565 scope.go:117] "RemoveContainer" containerID="6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a" Sep 07 10:13:36.335927 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:13:36.335907 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a\": container with ID starting with 6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a not found: ID does not exist" containerID="6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a" Sep 07 10:13:36.335978 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.335935 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a"} err="failed to get container status \"6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a\": rpc error: code = NotFound desc = could not find container \"6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a\": container with ID starting with 6ff3d9375ee29a9b2a291bcbbe398db174b78ce148830ecebd09b40574b8377a not found: ID does not exist" Sep 07 10:13:36.346838 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.346811 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c76479bc7-w98th"] Sep 07 10:13:36.350402 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.350380 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c76479bc7-w98th"] Sep 07 10:13:36.542459 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:36.542429 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bde39ddf-db09-473d-acfd-31dd8e2ef9be" path="/var/lib/kubelet/pods/bde39ddf-db09-473d-acfd-31dd8e2ef9be/volumes" Sep 07 10:13:42.354645 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:42.354557 2565 generic.go:356] "Generic (PLEG): container finished" podID="829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b" containerID="06f26fb4a674029e8fe234e323dcff857a41a24d9b1beb049b2320fa6c224b0d" exitCode=0 Sep 07 10:13:42.354645 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:42.354630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" event={"ID":"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b","Type":"ContainerDied","Data":"06f26fb4a674029e8fe234e323dcff857a41a24d9b1beb049b2320fa6c224b0d"} Sep 07 10:13:42.355080 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:42.354930 2565 scope.go:117] "RemoveContainer" containerID="06f26fb4a674029e8fe234e323dcff857a41a24d9b1beb049b2320fa6c224b0d" Sep 07 10:13:43.366034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:13:43.366002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n24qg" event={"ID":"829ad1ad-70d6-4f45-8dfc-e6428e2c5b5b","Type":"ContainerStarted","Data":"e8b004bc6d9cbc72ac27e14818b48bd196a407a91cbfcf888d7dbfad4111bb9f"} Sep 07 10:14:18.830462 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.830436 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:14:18.830831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.830720 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4496a2f9-ed1d-4e70-919d-b4968349702f" containerName="registry" Sep 07 10:14:18.830831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.830731 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4496a2f9-ed1d-4e70-919d-b4968349702f" containerName="registry" Sep 07 10:14:18.830831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.830755 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bde39ddf-db09-473d-acfd-31dd8e2ef9be" containerName="console" Sep 07 10:14:18.830831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.830760 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bde39ddf-db09-473d-acfd-31dd8e2ef9be" containerName="console" Sep 07 10:14:18.830831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.830809 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4496a2f9-ed1d-4e70-919d-b4968349702f" containerName="registry" Sep 07 10:14:18.830831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.830819 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bde39ddf-db09-473d-acfd-31dd8e2ef9be" containerName="console" Sep 07 10:14:18.833694 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.833671 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.836519 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.836499 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 07 10:14:18.836764 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.836744 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 07 10:14:18.836874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.836765 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 07 10:14:18.836874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.836765 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 07 10:14:18.836874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.836816 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 07 10:14:18.836874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.836867 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 07 10:14:18.837346 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.837328 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c5w5s\"" Sep 07 10:14:18.837431 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.837390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 07 10:14:18.837549 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.837536 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 07 10:14:18.842878 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.842858 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 07 10:14:18.847962 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.847943 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:14:18.981069 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981029 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981069 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b35295b-4705-4447-b827-810ac109521c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b35295b-4705-4447-b827-810ac109521c-config-out\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-web-config\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b35295b-4705-4447-b827-810ac109521c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b35295b-4705-4447-b827-810ac109521c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b35295b-4705-4447-b827-810ac109521c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv7lw\" (UniqueName: \"kubernetes.io/projected/7b35295b-4705-4447-b827-810ac109521c-kube-api-access-rv7lw\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:18.981480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:18.981420 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.082921 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.082802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.082921 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.082878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.082921 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.082901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b35295b-4705-4447-b827-810ac109521c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.082921 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.082924 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.082945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b35295b-4705-4447-b827-810ac109521c-config-out\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.082978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.083003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.083026 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-web-config\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.083064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b35295b-4705-4447-b827-810ac109521c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.083094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.083128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b35295b-4705-4447-b827-810ac109521c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.083151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b35295b-4705-4447-b827-810ac109521c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.083213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv7lw\" (UniqueName: \"kubernetes.io/projected/7b35295b-4705-4447-b827-810ac109521c-kube-api-access-rv7lw\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.083927 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.083464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b35295b-4705-4447-b827-810ac109521c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.084481 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.084451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b35295b-4705-4447-b827-810ac109521c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.086152 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.086126 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b35295b-4705-4447-b827-810ac109521c-config-out\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.086412 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.086382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.086609 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.086555 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.086678 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.086661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.086894 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.086872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.087316 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.087296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b35295b-4705-4447-b827-810ac109521c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.087389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.087316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.087389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.087369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.087517 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.087500 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b35295b-4705-4447-b827-810ac109521c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.088470 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.088444 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b35295b-4705-4447-b827-810ac109521c-web-config\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.091444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.091427 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv7lw\" (UniqueName: \"kubernetes.io/projected/7b35295b-4705-4447-b827-810ac109521c-kube-api-access-rv7lw\") pod \"alertmanager-main-0\" (UID: \"7b35295b-4705-4447-b827-810ac109521c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.144370 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.144332 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:14:19.421686 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.421660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:14:19.422114 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:14:19.422091 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b35295b_4705_4447_b827_810ac109521c.slice/crio-6de47741df8941f7127745f01a2a69b40bb4453131945d6b43d0a3f8c4b92021 WatchSource:0}: Error finding container 6de47741df8941f7127745f01a2a69b40bb4453131945d6b43d0a3f8c4b92021: Status 404 returned error can't find the container with id 6de47741df8941f7127745f01a2a69b40bb4453131945d6b43d0a3f8c4b92021 Sep 07 10:14:19.510823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.510791 2565 generic.go:356] "Generic (PLEG): container finished" podID="7b35295b-4705-4447-b827-810ac109521c" containerID="2f7d3995afc2389cefd2809502ee7652c19c2e7fd6c9e5695727636366a86aea" exitCode=0 Sep 07 10:14:19.510923 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.510867 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b35295b-4705-4447-b827-810ac109521c","Type":"ContainerDied","Data":"2f7d3995afc2389cefd2809502ee7652c19c2e7fd6c9e5695727636366a86aea"} Sep 07 10:14:19.510969 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:19.510935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b35295b-4705-4447-b827-810ac109521c","Type":"ContainerStarted","Data":"6de47741df8941f7127745f01a2a69b40bb4453131945d6b43d0a3f8c4b92021"} Sep 07 10:14:21.523354 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:21.523272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b35295b-4705-4447-b827-810ac109521c","Type":"ContainerStarted","Data":"e89e43e4134d53890e2b0c2ef9bbeaf47fff03e7fe4ed2c6fcabba5536184008"} Sep 07 10:14:21.523354 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:21.523337 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b35295b-4705-4447-b827-810ac109521c","Type":"ContainerStarted","Data":"9a1ef6f31c855c7132bd23a3acec345dbefd1a3e0d69955b0c75b84c16a56b45"} Sep 07 10:14:21.523740 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:21.523362 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b35295b-4705-4447-b827-810ac109521c","Type":"ContainerStarted","Data":"5f76e99fbbcd2d203aed8f8c5f5dcf7e526c50c5d1a188bbd5239ac5e76698fc"} Sep 07 10:14:21.523740 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:21.523386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b35295b-4705-4447-b827-810ac109521c","Type":"ContainerStarted","Data":"6c8b52a2a6b074b373c8ed3761bfa2c6d55c684069ea4a71d9ddac55801b0308"} Sep 07 10:14:21.523740 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:21.523409 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b35295b-4705-4447-b827-810ac109521c","Type":"ContainerStarted","Data":"331dcb1945aa2125961dbffd7a832abef1783d04fbd9b5ef3c38ad470ffe6450"} Sep 07 10:14:22.530301 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:22.530267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b35295b-4705-4447-b827-810ac109521c","Type":"ContainerStarted","Data":"8007af106a0a823d248e9542a7a076caa3ade7dae8c72bbcabbafde94c1825d9"} Sep 07 10:14:22.557308 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:22.557258 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.308524506 podStartE2EDuration="4.557242548s" totalImagesPullingTime="2.233660489s" totalInitContainerRuntime="15.057553ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:14:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:14:22.555064924 +0000 UTC m=+184.541345259" watchObservedRunningTime="2026-09-07 10:14:22.557242548 +0000 UTC m=+184.543522882" Sep 07 10:14:31.044825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.044781 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fbfc8d6f6-z4t2q"] Sep 07 10:14:31.047724 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.047703 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.054500 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.054481 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 07 10:14:31.054606 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.054584 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ndkq9\"" Sep 07 10:14:31.055646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.055621 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 07 10:14:31.055646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.055629 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 07 10:14:31.055789 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.055666 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 07 10:14:31.056796 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.056778 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 07 10:14:31.063498 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.063480 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 07 10:14:31.066325 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.066308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbfc8d6f6-z4t2q"] Sep 07 10:14:31.082703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.082679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-serving-cert\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.082798 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.082708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-oauth-serving-cert\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.082798 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.082761 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnwhk\" (UniqueName: \"kubernetes.io/projected/bc198495-2f01-449a-a864-f097584b4fb9-kube-api-access-fnwhk\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.082798 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.082784 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-oauth-config\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.082920 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.082806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-trusted-ca-bundle\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.082920 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.082821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-service-ca\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.082920 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.082885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-console-config\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.183777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.183745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-oauth-serving-cert\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.183941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.183807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnwhk\" (UniqueName: \"kubernetes.io/projected/bc198495-2f01-449a-a864-f097584b4fb9-kube-api-access-fnwhk\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.183941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.183834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-oauth-config\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.183941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.183859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-trusted-ca-bundle\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.183941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.183882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-service-ca\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.183941 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.183939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-console-config\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.184206 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.183979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-serving-cert\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.184613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.184577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-oauth-serving-cert\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.184723 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.184663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-console-config\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.184723 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.184676 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-service-ca\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.184855 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.184771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-trusted-ca-bundle\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.186510 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.186491 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-serving-cert\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.186657 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.186641 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-oauth-config\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.192387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.192361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnwhk\" (UniqueName: \"kubernetes.io/projected/bc198495-2f01-449a-a864-f097584b4fb9-kube-api-access-fnwhk\") pod \"console-fbfc8d6f6-z4t2q\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.356734 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.356630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:31.629021 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:31.628995 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbfc8d6f6-z4t2q"] Sep 07 10:14:31.631646 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:14:31.631603 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc198495_2f01_449a_a864_f097584b4fb9.slice/crio-a736c9292c02cfd5f0eaed1b06357540bfebbb0e9d13699de979b1d3b8cacc50 WatchSource:0}: Error finding container a736c9292c02cfd5f0eaed1b06357540bfebbb0e9d13699de979b1d3b8cacc50: Status 404 returned error can't find the container with id a736c9292c02cfd5f0eaed1b06357540bfebbb0e9d13699de979b1d3b8cacc50 Sep 07 10:14:32.570297 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:32.570261 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbfc8d6f6-z4t2q" event={"ID":"bc198495-2f01-449a-a864-f097584b4fb9","Type":"ContainerStarted","Data":"161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c"} Sep 07 10:14:32.570667 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:32.570308 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbfc8d6f6-z4t2q" event={"ID":"bc198495-2f01-449a-a864-f097584b4fb9","Type":"ContainerStarted","Data":"a736c9292c02cfd5f0eaed1b06357540bfebbb0e9d13699de979b1d3b8cacc50"} Sep 07 10:14:32.589230 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:32.589183 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fbfc8d6f6-z4t2q" podStartSLOduration=1.589154265 podStartE2EDuration="1.589154265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:14:32.587432657 +0000 UTC m=+194.573712990" watchObservedRunningTime="2026-09-07 10:14:32.589154265 +0000 UTC m=+194.575434598" Sep 07 10:14:41.357109 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:41.357073 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:41.357533 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:41.357135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:41.361929 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:41.361907 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:14:41.610451 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:14:41.610365 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:15:33.079354 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.079318 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp"] Sep 07 10:15:33.083903 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.083883 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.086649 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.086619 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 07 10:15:33.086649 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.086642 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 07 10:15:33.087465 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.087449 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m28xb\"" Sep 07 10:15:33.091673 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.091650 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp"] Sep 07 10:15:33.218520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.218490 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twwc7\" (UniqueName: \"kubernetes.io/projected/8de5c174-d71b-46ae-8353-fb459e6092ce-kube-api-access-twwc7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.218704 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.218540 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.218704 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.218624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.319483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.319454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.319627 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.319513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.319627 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.319537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twwc7\" (UniqueName: \"kubernetes.io/projected/8de5c174-d71b-46ae-8353-fb459e6092ce-kube-api-access-twwc7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.319871 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.319854 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.319909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.319875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.327374 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.327353 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twwc7\" (UniqueName: \"kubernetes.io/projected/8de5c174-d71b-46ae-8353-fb459e6092ce-kube-api-access-twwc7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.393659 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.393591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:33.670875 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.670851 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp"] Sep 07 10:15:33.672560 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:15:33.672537 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de5c174_d71b_46ae_8353_fb459e6092ce.slice/crio-a763dda75216f77d6ddb2e62f9c0df3d95345fd029a44e6a19bb1a57301eba61 WatchSource:0}: Error finding container a763dda75216f77d6ddb2e62f9c0df3d95345fd029a44e6a19bb1a57301eba61: Status 404 returned error can't find the container with id a763dda75216f77d6ddb2e62f9c0df3d95345fd029a44e6a19bb1a57301eba61 Sep 07 10:15:33.817546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:33.817510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" event={"ID":"8de5c174-d71b-46ae-8353-fb459e6092ce","Type":"ContainerStarted","Data":"a763dda75216f77d6ddb2e62f9c0df3d95345fd029a44e6a19bb1a57301eba61"} Sep 07 10:15:39.855985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:39.855954 2565 generic.go:356] "Generic (PLEG): container finished" podID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerID="23d51bcde89b069ded282e41d72008ec05c62033a5faee0b3a88f577409508c4" exitCode=0 Sep 07 10:15:39.856390 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:39.856041 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" event={"ID":"8de5c174-d71b-46ae-8353-fb459e6092ce","Type":"ContainerDied","Data":"23d51bcde89b069ded282e41d72008ec05c62033a5faee0b3a88f577409508c4"} Sep 07 10:15:42.872872 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:42.872835 2565 generic.go:356] "Generic (PLEG): container finished" podID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerID="e50957808429d5bf62b5bf08d892dc3edb6a582075e07c2a6987e30db3f18211" exitCode=0 Sep 07 10:15:42.873292 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:42.872890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" event={"ID":"8de5c174-d71b-46ae-8353-fb459e6092ce","Type":"ContainerDied","Data":"e50957808429d5bf62b5bf08d892dc3edb6a582075e07c2a6987e30db3f18211"} Sep 07 10:15:49.907439 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:49.907406 2565 generic.go:356] "Generic (PLEG): container finished" podID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerID="53817514fedf28e3358e25482a8f22432c5cb966b29ed1d2db4e40338bb1325c" exitCode=0 Sep 07 10:15:49.907901 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:49.907500 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" event={"ID":"8de5c174-d71b-46ae-8353-fb459e6092ce","Type":"ContainerDied","Data":"53817514fedf28e3358e25482a8f22432c5cb966b29ed1d2db4e40338bb1325c"} Sep 07 10:15:51.035458 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.035433 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:51.184345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.184250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twwc7\" (UniqueName: \"kubernetes.io/projected/8de5c174-d71b-46ae-8353-fb459e6092ce-kube-api-access-twwc7\") pod \"8de5c174-d71b-46ae-8353-fb459e6092ce\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " Sep 07 10:15:51.184345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.184296 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-util\") pod \"8de5c174-d71b-46ae-8353-fb459e6092ce\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " Sep 07 10:15:51.184345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.184315 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-bundle\") pod \"8de5c174-d71b-46ae-8353-fb459e6092ce\" (UID: \"8de5c174-d71b-46ae-8353-fb459e6092ce\") " Sep 07 10:15:51.184944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.184918 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-bundle" (OuterVolumeSpecName: "bundle") pod "8de5c174-d71b-46ae-8353-fb459e6092ce" (UID: "8de5c174-d71b-46ae-8353-fb459e6092ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:15:51.186639 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.186613 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de5c174-d71b-46ae-8353-fb459e6092ce-kube-api-access-twwc7" (OuterVolumeSpecName: "kube-api-access-twwc7") pod "8de5c174-d71b-46ae-8353-fb459e6092ce" (UID: "8de5c174-d71b-46ae-8353-fb459e6092ce"). InnerVolumeSpecName "kube-api-access-twwc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:15:51.188638 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.188617 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-util" (OuterVolumeSpecName: "util") pod "8de5c174-d71b-46ae-8353-fb459e6092ce" (UID: "8de5c174-d71b-46ae-8353-fb459e6092ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:15:51.285131 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.285100 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twwc7\" (UniqueName: \"kubernetes.io/projected/8de5c174-d71b-46ae-8353-fb459e6092ce-kube-api-access-twwc7\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:15:51.285131 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.285128 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-util\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:15:51.285131 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.285139 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8de5c174-d71b-46ae-8353-fb459e6092ce-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:15:51.918748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.918714 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" Sep 07 10:15:51.918748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.918724 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w2lpp" event={"ID":"8de5c174-d71b-46ae-8353-fb459e6092ce","Type":"ContainerDied","Data":"a763dda75216f77d6ddb2e62f9c0df3d95345fd029a44e6a19bb1a57301eba61"} Sep 07 10:15:51.918960 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:15:51.918773 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a763dda75216f77d6ddb2e62f9c0df3d95345fd029a44e6a19bb1a57301eba61" Sep 07 10:16:00.090180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.090121 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-swsrc"] Sep 07 10:16:00.090626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.090488 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerName="pull" Sep 07 10:16:00.090626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.090501 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerName="pull" Sep 07 10:16:00.090626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.090522 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerName="util" Sep 07 10:16:00.090626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.090527 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerName="util" Sep 07 10:16:00.090626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.090533 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerName="extract" Sep 07 10:16:00.090626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.090539 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerName="extract" Sep 07 10:16:00.090626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.090611 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de5c174-d71b-46ae-8353-fb459e6092ce" containerName="extract" Sep 07 10:16:00.134374 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.134336 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-swsrc"] Sep 07 10:16:00.134545 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.134435 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:00.137181 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.137138 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 07 10:16:00.137317 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.137259 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 07 10:16:00.159701 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.159666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec85c73-0279-4cc2-bc17-829a82aee2de-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-swsrc\" (UID: \"0ec85c73-0279-4cc2-bc17-829a82aee2de\") " pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:00.159863 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.159704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gkqt\" (UniqueName: \"kubernetes.io/projected/0ec85c73-0279-4cc2-bc17-829a82aee2de-kube-api-access-7gkqt\") pod \"cert-manager-webhook-6486ffd997-swsrc\" (UID: \"0ec85c73-0279-4cc2-bc17-829a82aee2de\") " pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:00.260493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.260458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec85c73-0279-4cc2-bc17-829a82aee2de-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-swsrc\" (UID: \"0ec85c73-0279-4cc2-bc17-829a82aee2de\") " pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:00.260658 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.260503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gkqt\" (UniqueName: \"kubernetes.io/projected/0ec85c73-0279-4cc2-bc17-829a82aee2de-kube-api-access-7gkqt\") pod \"cert-manager-webhook-6486ffd997-swsrc\" (UID: \"0ec85c73-0279-4cc2-bc17-829a82aee2de\") " pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:00.270128 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.270095 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec85c73-0279-4cc2-bc17-829a82aee2de-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-swsrc\" (UID: \"0ec85c73-0279-4cc2-bc17-829a82aee2de\") " pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:00.272092 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.272070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gkqt\" (UniqueName: \"kubernetes.io/projected/0ec85c73-0279-4cc2-bc17-829a82aee2de-kube-api-access-7gkqt\") pod \"cert-manager-webhook-6486ffd997-swsrc\" (UID: \"0ec85c73-0279-4cc2-bc17-829a82aee2de\") " pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:00.459875 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.459788 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:00.759202 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.759174 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-swsrc"] Sep 07 10:16:00.762927 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:16:00.762894 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ec85c73_0279_4cc2_bc17_829a82aee2de.slice/crio-6cb5de00d022ec4bc8bc581c289737d67e1edf50036718b09f5bd27b0d73cc48 WatchSource:0}: Error finding container 6cb5de00d022ec4bc8bc581c289737d67e1edf50036718b09f5bd27b0d73cc48: Status 404 returned error can't find the container with id 6cb5de00d022ec4bc8bc581c289737d67e1edf50036718b09f5bd27b0d73cc48 Sep 07 10:16:00.960609 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:00.960570 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" event={"ID":"0ec85c73-0279-4cc2-bc17-829a82aee2de","Type":"ContainerStarted","Data":"6cb5de00d022ec4bc8bc581c289737d67e1edf50036718b09f5bd27b0d73cc48"} Sep 07 10:16:03.977200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:03.977169 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" event={"ID":"0ec85c73-0279-4cc2-bc17-829a82aee2de","Type":"ContainerStarted","Data":"5d25c2dce3607e5166dd98f1462357ecc08f118d26eeab7e40e2967a50762fa5"} Sep 07 10:16:03.977568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:03.977252 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:03.995110 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:03.995051 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" podStartSLOduration=0.968839836 podStartE2EDuration="3.995032841s" totalImagesPullingTime="3.026193005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:16:03.994314736 +0000 UTC m=+285.980595070" watchObservedRunningTime="2026-09-07 10:16:03.995032841 +0000 UTC m=+285.981313174" Sep 07 10:16:09.983811 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:09.983774 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-swsrc" Sep 07 10:16:18.446144 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:18.446112 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:16:18.450148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:18.450123 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:16:18.458862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:18.458840 2565 kubelet.go:1648] "Image garbage collection succeeded" Sep 07 10:16:37.676717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.676680 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9"] Sep 07 10:16:37.684042 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.684013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.686884 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.686862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 07 10:16:37.687396 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.687372 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9"] Sep 07 10:16:37.687866 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.687830 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 07 10:16:37.687974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.687908 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m28xb\"" Sep 07 10:16:37.756765 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.756736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.756909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.756771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.756909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.756820 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7t8p\" (UniqueName: \"kubernetes.io/projected/4b3958ab-5bcf-4f3a-9078-02fd2813198d-kube-api-access-q7t8p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.857814 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.857777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.857996 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.857863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7t8p\" (UniqueName: \"kubernetes.io/projected/4b3958ab-5bcf-4f3a-9078-02fd2813198d-kube-api-access-q7t8p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.857996 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.857917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.858188 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.858141 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.858270 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.858244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.866333 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.866316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7t8p\" (UniqueName: \"kubernetes.io/projected/4b3958ab-5bcf-4f3a-9078-02fd2813198d-kube-api-access-q7t8p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:37.994510 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:37.994476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:38.266541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:38.266509 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9"] Sep 07 10:16:38.269341 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:16:38.269309 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3958ab_5bcf_4f3a_9078_02fd2813198d.slice/crio-4d03f85488e87e3fac65193e250f9c0e063f5f2be41ca4d250f4c2ad8a769540 WatchSource:0}: Error finding container 4d03f85488e87e3fac65193e250f9c0e063f5f2be41ca4d250f4c2ad8a769540: Status 404 returned error can't find the container with id 4d03f85488e87e3fac65193e250f9c0e063f5f2be41ca4d250f4c2ad8a769540 Sep 07 10:16:39.134091 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:39.134004 2565 generic.go:356] "Generic (PLEG): container finished" podID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerID="025ec032fbb4b5cd6215b853d9ca53b6532b4d4b15094163869965be89ba745f" exitCode=0 Sep 07 10:16:39.134466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:39.134085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" event={"ID":"4b3958ab-5bcf-4f3a-9078-02fd2813198d","Type":"ContainerDied","Data":"025ec032fbb4b5cd6215b853d9ca53b6532b4d4b15094163869965be89ba745f"} Sep 07 10:16:39.134466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:39.134128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" event={"ID":"4b3958ab-5bcf-4f3a-9078-02fd2813198d","Type":"ContainerStarted","Data":"4d03f85488e87e3fac65193e250f9c0e063f5f2be41ca4d250f4c2ad8a769540"} Sep 07 10:16:39.134588 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:39.134574 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:16:40.141381 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:40.141294 2565 generic.go:356] "Generic (PLEG): container finished" podID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerID="fd8734094fe53782566950a7ded31f4cd1d0cef388866a5df5bf2ff0149eb592" exitCode=0 Sep 07 10:16:40.141381 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:40.141360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" event={"ID":"4b3958ab-5bcf-4f3a-9078-02fd2813198d","Type":"ContainerDied","Data":"fd8734094fe53782566950a7ded31f4cd1d0cef388866a5df5bf2ff0149eb592"} Sep 07 10:16:41.154268 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:41.154232 2565 generic.go:356] "Generic (PLEG): container finished" podID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerID="02e41ea180544940fb135eddcee7f859c9a2f94791dfcdc5dc038a934c26e15c" exitCode=0 Sep 07 10:16:41.154656 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:41.154314 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" event={"ID":"4b3958ab-5bcf-4f3a-9078-02fd2813198d","Type":"ContainerDied","Data":"02e41ea180544940fb135eddcee7f859c9a2f94791dfcdc5dc038a934c26e15c"} Sep 07 10:16:42.285261 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.285235 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:42.397972 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.397941 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-util\") pod \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " Sep 07 10:16:42.398149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.398004 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7t8p\" (UniqueName: \"kubernetes.io/projected/4b3958ab-5bcf-4f3a-9078-02fd2813198d-kube-api-access-q7t8p\") pod \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " Sep 07 10:16:42.398149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.398100 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-bundle\") pod \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\" (UID: \"4b3958ab-5bcf-4f3a-9078-02fd2813198d\") " Sep 07 10:16:42.398804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.398769 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-bundle" (OuterVolumeSpecName: "bundle") pod "4b3958ab-5bcf-4f3a-9078-02fd2813198d" (UID: "4b3958ab-5bcf-4f3a-9078-02fd2813198d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:16:42.400330 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.400305 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3958ab-5bcf-4f3a-9078-02fd2813198d-kube-api-access-q7t8p" (OuterVolumeSpecName: "kube-api-access-q7t8p") pod "4b3958ab-5bcf-4f3a-9078-02fd2813198d" (UID: "4b3958ab-5bcf-4f3a-9078-02fd2813198d"). InnerVolumeSpecName "kube-api-access-q7t8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:16:42.403950 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.403912 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-util" (OuterVolumeSpecName: "util") pod "4b3958ab-5bcf-4f3a-9078-02fd2813198d" (UID: "4b3958ab-5bcf-4f3a-9078-02fd2813198d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:16:42.499224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.499193 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:16:42.499224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.499221 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3958ab-5bcf-4f3a-9078-02fd2813198d-util\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:16:42.499403 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:42.499232 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7t8p\" (UniqueName: \"kubernetes.io/projected/4b3958ab-5bcf-4f3a-9078-02fd2813198d-kube-api-access-q7t8p\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:16:43.165964 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:43.165937 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" Sep 07 10:16:43.165964 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:43.165945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqq8t9" event={"ID":"4b3958ab-5bcf-4f3a-9078-02fd2813198d","Type":"ContainerDied","Data":"4d03f85488e87e3fac65193e250f9c0e063f5f2be41ca4d250f4c2ad8a769540"} Sep 07 10:16:43.166187 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:43.165983 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d03f85488e87e3fac65193e250f9c0e063f5f2be41ca4d250f4c2ad8a769540" Sep 07 10:16:49.239992 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.239959 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj"] Sep 07 10:16:49.240518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.240303 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerName="util" Sep 07 10:16:49.240518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.240315 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerName="util" Sep 07 10:16:49.240518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.240330 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerName="extract" Sep 07 10:16:49.240518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.240335 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerName="extract" Sep 07 10:16:49.240518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.240344 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerName="pull" Sep 07 10:16:49.240518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.240350 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerName="pull" Sep 07 10:16:49.240518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.240432 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b3958ab-5bcf-4f3a-9078-02fd2813198d" containerName="extract" Sep 07 10:16:49.243582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.243567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.246349 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.246328 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 07 10:16:49.247759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.247729 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 07 10:16:49.247759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.247750 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:16:49.247919 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.247778 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 07 10:16:49.247919 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.247782 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 07 10:16:49.247919 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.247828 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-w9qqf\"" Sep 07 10:16:49.253138 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.253118 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj"] Sep 07 10:16:49.359310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.359270 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d481477b-c29f-45de-9658-825a4fcc5676-manager-config\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.359310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.359309 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d481477b-c29f-45de-9658-825a4fcc5676-metrics-cert\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.359538 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.359382 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gmg8\" (UniqueName: \"kubernetes.io/projected/d481477b-c29f-45de-9658-825a4fcc5676-kube-api-access-5gmg8\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.359538 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.359408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d481477b-c29f-45de-9658-825a4fcc5676-cert\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.460234 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.460197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmg8\" (UniqueName: \"kubernetes.io/projected/d481477b-c29f-45de-9658-825a4fcc5676-kube-api-access-5gmg8\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.460234 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.460242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d481477b-c29f-45de-9658-825a4fcc5676-cert\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.460419 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.460293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d481477b-c29f-45de-9658-825a4fcc5676-manager-config\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.460419 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.460309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d481477b-c29f-45de-9658-825a4fcc5676-metrics-cert\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.460961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.460940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d481477b-c29f-45de-9658-825a4fcc5676-manager-config\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.462994 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.462977 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d481477b-c29f-45de-9658-825a4fcc5676-metrics-cert\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.463054 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.463020 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d481477b-c29f-45de-9658-825a4fcc5676-cert\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.469206 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.469181 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmg8\" (UniqueName: \"kubernetes.io/projected/d481477b-c29f-45de-9658-825a4fcc5676-kube-api-access-5gmg8\") pod \"lws-controller-manager-6c54b9f868-s8nvj\" (UID: \"d481477b-c29f-45de-9658-825a4fcc5676\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.553545 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.553460 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:49.833086 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:49.833062 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj"] Sep 07 10:16:49.835013 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:16:49.834986 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd481477b_c29f_45de_9658_825a4fcc5676.slice/crio-45892a30bea38fa71d057ad88ddd2b1e284aa2fe2fab8e1ec6e43a56fb8eaaa6 WatchSource:0}: Error finding container 45892a30bea38fa71d057ad88ddd2b1e284aa2fe2fab8e1ec6e43a56fb8eaaa6: Status 404 returned error can't find the container with id 45892a30bea38fa71d057ad88ddd2b1e284aa2fe2fab8e1ec6e43a56fb8eaaa6 Sep 07 10:16:50.198900 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:50.198818 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" event={"ID":"d481477b-c29f-45de-9658-825a4fcc5676","Type":"ContainerStarted","Data":"45892a30bea38fa71d057ad88ddd2b1e284aa2fe2fab8e1ec6e43a56fb8eaaa6"} Sep 07 10:16:53.222194 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:53.222139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" event={"ID":"d481477b-c29f-45de-9658-825a4fcc5676","Type":"ContainerStarted","Data":"5829b5c9677c923a755ce63d8d8eb63f7d7968af1368d43b6d223da18143f4a5"} Sep 07 10:16:53.222564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:53.222277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:16:53.239076 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:53.239020 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" podStartSLOduration=1.837655523 podStartE2EDuration="4.239001348s" totalImagesPullingTime="2.401345825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:16:53.237944755 +0000 UTC m=+335.224225088" watchObservedRunningTime="2026-09-07 10:16:53.239001348 +0000 UTC m=+335.225281683" Sep 07 10:16:56.267903 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.267860 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k"] Sep 07 10:16:56.271531 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.271508 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.274545 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.274520 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 07 10:16:56.274669 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.274570 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 07 10:16:56.274669 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.274570 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 07 10:16:56.274669 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.274655 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q77qk\"" Sep 07 10:16:56.274823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.274814 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 07 10:16:56.288795 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.288763 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k"] Sep 07 10:16:56.421186 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.421128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9266e4c7-c4e2-467b-aa93-70fc1b738732-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.421364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.421259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9266e4c7-c4e2-467b-aa93-70fc1b738732-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.421364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.421333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9266e4c7-c4e2-467b-aa93-70fc1b738732-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.421438 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.421374 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9266e4c7-c4e2-467b-aa93-70fc1b738732-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.421438 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.421417 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmtr\" (UniqueName: \"kubernetes.io/projected/9266e4c7-c4e2-467b-aa93-70fc1b738732-kube-api-access-4dmtr\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.522055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.521963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9266e4c7-c4e2-467b-aa93-70fc1b738732-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.522055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.522006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9266e4c7-c4e2-467b-aa93-70fc1b738732-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.522055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.522039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmtr\" (UniqueName: \"kubernetes.io/projected/9266e4c7-c4e2-467b-aa93-70fc1b738732-kube-api-access-4dmtr\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.522332 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.522091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9266e4c7-c4e2-467b-aa93-70fc1b738732-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.522332 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.522152 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9266e4c7-c4e2-467b-aa93-70fc1b738732-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.522495 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.522456 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9266e4c7-c4e2-467b-aa93-70fc1b738732-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.524651 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.524622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9266e4c7-c4e2-467b-aa93-70fc1b738732-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.524736 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.524713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9266e4c7-c4e2-467b-aa93-70fc1b738732-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.524789 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.524775 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9266e4c7-c4e2-467b-aa93-70fc1b738732-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.530363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.530338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmtr\" (UniqueName: \"kubernetes.io/projected/9266e4c7-c4e2-467b-aa93-70fc1b738732-kube-api-access-4dmtr\") pod \"opendatahub-operator-controller-manager-685c7f779d-krp6k\" (UID: \"9266e4c7-c4e2-467b-aa93-70fc1b738732\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.582114 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.582075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:16:56.859882 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:56.859769 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k"] Sep 07 10:16:56.862695 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:16:56.862669 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9266e4c7_c4e2_467b_aa93_70fc1b738732.slice/crio-d514714d08a741b2df454494138cd479ccb76682dd3bfe6b1a753a668b0b844b WatchSource:0}: Error finding container d514714d08a741b2df454494138cd479ccb76682dd3bfe6b1a753a668b0b844b: Status 404 returned error can't find the container with id d514714d08a741b2df454494138cd479ccb76682dd3bfe6b1a753a668b0b844b Sep 07 10:16:57.249794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:16:57.249754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" event={"ID":"9266e4c7-c4e2-467b-aa93-70fc1b738732","Type":"ContainerStarted","Data":"d514714d08a741b2df454494138cd479ccb76682dd3bfe6b1a753a668b0b844b"} Sep 07 10:17:00.265919 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:00.265874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" event={"ID":"9266e4c7-c4e2-467b-aa93-70fc1b738732","Type":"ContainerStarted","Data":"37529c64ea0a4a665b1145c1138286b0c4aeb93ad93fe86214e0f953cb1370d9"} Sep 07 10:17:01.271581 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:01.271553 2565 generic.go:356] "Generic (PLEG): container finished" podID="9266e4c7-c4e2-467b-aa93-70fc1b738732" containerID="37529c64ea0a4a665b1145c1138286b0c4aeb93ad93fe86214e0f953cb1370d9" exitCode=0 Sep 07 10:17:01.271987 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:01.271627 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" event={"ID":"9266e4c7-c4e2-467b-aa93-70fc1b738732","Type":"ContainerDied","Data":"37529c64ea0a4a665b1145c1138286b0c4aeb93ad93fe86214e0f953cb1370d9"} Sep 07 10:17:02.279293 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:02.279257 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" event={"ID":"9266e4c7-c4e2-467b-aa93-70fc1b738732","Type":"ContainerStarted","Data":"79505bb55818060dc6aeda2eca36f6e9bac2756febe5fa913d5d11528d4f739d"} Sep 07 10:17:02.279702 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:02.279379 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:17:02.301292 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:02.301236 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" podStartSLOduration=1.9348150290000001 podStartE2EDuration="6.301218753s" totalImagesPullingTime="3.310912864s" totalInitContainerRuntime="1.05549086s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:02.299823344 +0000 UTC m=+344.286103679" watchObservedRunningTime="2026-09-07 10:17:02.301218753 +0000 UTC m=+344.287499086" Sep 07 10:17:04.232636 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:04.232608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-s8nvj" Sep 07 10:17:13.287024 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:13.286935 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-krp6k" Sep 07 10:17:23.304571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.304533 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88"] Sep 07 10:17:23.314943 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.314916 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.317717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.317642 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 07 10:17:23.317980 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.317960 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zl2mp\"" Sep 07 10:17:23.318771 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.318750 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88"] Sep 07 10:17:23.343712 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.343682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.343851 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.343735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.343851 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.343762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzsbn\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-kube-api-access-jzsbn\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.343851 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.343804 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.343851 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.343828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.344072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.343852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.344072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.343874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.344072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.343972 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.344072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.344022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c684cdc5-ab92-4236-b1d1-6df043a2f531-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzsbn\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-kube-api-access-jzsbn\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445681 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445681 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c684cdc5-ab92-4236-b1d1-6df043a2f531-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445681 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445889 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445752 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445889 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445813 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.445961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.445946 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.446333 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.446309 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c684cdc5-ab92-4236-b1d1-6df043a2f531-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.448313 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.448286 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.448624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.448605 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.453322 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.453294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.453434 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.453422 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzsbn\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-kube-api-access-jzsbn\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-28v88\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.627072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.626997 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:23.630742 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.630715 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv"] Sep 07 10:17:23.634260 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.634236 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401435cb-b468-417d-b7b5-d98787fa98f0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647500 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401435cb-b468-417d-b7b5-d98787fa98f0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647659 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647659 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxcdv\" (UniqueName: \"kubernetes.io/projected/401435cb-b468-417d-b7b5-d98787fa98f0-kube-api-access-jxcdv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647659 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401435cb-b468-417d-b7b5-d98787fa98f0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647835 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.647973 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.647953 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv"] Sep 07 10:17:23.748862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.748831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.748862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.748863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401435cb-b468-417d-b7b5-d98787fa98f0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.748891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.748920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.748946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401435cb-b468-417d-b7b5-d98787fa98f0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.748979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.749018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxcdv\" (UniqueName: \"kubernetes.io/projected/401435cb-b468-417d-b7b5-d98787fa98f0-kube-api-access-jxcdv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.749039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401435cb-b468-417d-b7b5-d98787fa98f0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749089 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.749067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.749322 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.749370 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.749495 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749705 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.749596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.749778 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.749760 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401435cb-b468-417d-b7b5-d98787fa98f0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.751397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.751378 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401435cb-b468-417d-b7b5-d98787fa98f0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.751602 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.751587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401435cb-b468-417d-b7b5-d98787fa98f0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.756912 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.756889 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401435cb-b468-417d-b7b5-d98787fa98f0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.757021 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.756997 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxcdv\" (UniqueName: \"kubernetes.io/projected/401435cb-b468-417d-b7b5-d98787fa98f0-kube-api-access-jxcdv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv\" (UID: \"401435cb-b468-417d-b7b5-d98787fa98f0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:23.906904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.906878 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88"] Sep 07 10:17:23.909537 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:17:23.909513 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc684cdc5_ab92_4236_b1d1_6df043a2f531.slice/crio-f03e3596438f56bbeac2e1bcec91b8d37ad341abd97101e94948a851628c4782 WatchSource:0}: Error finding container f03e3596438f56bbeac2e1bcec91b8d37ad341abd97101e94948a851628c4782: Status 404 returned error can't find the container with id f03e3596438f56bbeac2e1bcec91b8d37ad341abd97101e94948a851628c4782 Sep 07 10:17:23.962098 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:23.962063 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:24.236312 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:24.236289 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv"] Sep 07 10:17:24.238090 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:17:24.238065 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401435cb_b468_417d_b7b5_d98787fa98f0.slice/crio-1d9c7664910ae8f406b176414df2f6a0934714c5f3d94400c5843112fbefe89c WatchSource:0}: Error finding container 1d9c7664910ae8f406b176414df2f6a0934714c5f3d94400c5843112fbefe89c: Status 404 returned error can't find the container with id 1d9c7664910ae8f406b176414df2f6a0934714c5f3d94400c5843112fbefe89c Sep 07 10:17:24.387226 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:24.387187 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" event={"ID":"c684cdc5-ab92-4236-b1d1-6df043a2f531","Type":"ContainerStarted","Data":"f03e3596438f56bbeac2e1bcec91b8d37ad341abd97101e94948a851628c4782"} Sep 07 10:17:24.388832 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:24.388808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" event={"ID":"401435cb-b468-417d-b7b5-d98787fa98f0","Type":"ContainerStarted","Data":"1d9c7664910ae8f406b176414df2f6a0934714c5f3d94400c5843112fbefe89c"} Sep 07 10:17:26.244655 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.244619 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zhpzs"] Sep 07 10:17:26.252258 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.252225 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:26.255269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.255244 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-npvtn\"" Sep 07 10:17:26.255777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.255750 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 07 10:17:26.267143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.267074 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zhpzs"] Sep 07 10:17:26.373688 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.373647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07a4616c-f300-418e-8027-6b8efc43c741-cert\") pod \"odh-model-controller-7998c67dc4-zhpzs\" (UID: \"07a4616c-f300-418e-8027-6b8efc43c741\") " pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:26.373880 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.373792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdmx8\" (UniqueName: \"kubernetes.io/projected/07a4616c-f300-418e-8027-6b8efc43c741-kube-api-access-gdmx8\") pod \"odh-model-controller-7998c67dc4-zhpzs\" (UID: \"07a4616c-f300-418e-8027-6b8efc43c741\") " pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:26.475243 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.475204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07a4616c-f300-418e-8027-6b8efc43c741-cert\") pod \"odh-model-controller-7998c67dc4-zhpzs\" (UID: \"07a4616c-f300-418e-8027-6b8efc43c741\") " pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:26.475437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.475339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdmx8\" (UniqueName: \"kubernetes.io/projected/07a4616c-f300-418e-8027-6b8efc43c741-kube-api-access-gdmx8\") pod \"odh-model-controller-7998c67dc4-zhpzs\" (UID: \"07a4616c-f300-418e-8027-6b8efc43c741\") " pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:26.478308 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.478280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07a4616c-f300-418e-8027-6b8efc43c741-cert\") pod \"odh-model-controller-7998c67dc4-zhpzs\" (UID: \"07a4616c-f300-418e-8027-6b8efc43c741\") " pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:26.483884 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.483858 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdmx8\" (UniqueName: \"kubernetes.io/projected/07a4616c-f300-418e-8027-6b8efc43c741-kube-api-access-gdmx8\") pod \"odh-model-controller-7998c67dc4-zhpzs\" (UID: \"07a4616c-f300-418e-8027-6b8efc43c741\") " pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:26.569133 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.569050 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:26.956866 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.956806 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:17:26.957033 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.956919 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:17:26.957033 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.956990 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:17:26.962794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.962760 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:17:26.962906 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.962835 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:17:26.962906 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:26.962877 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:17:27.143679 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:27.143521 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zhpzs"] Sep 07 10:17:27.159967 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:17:27.159928 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a4616c_f300_418e_8027_6b8efc43c741.slice/crio-7620b71347287a0eaae4186064a3b4f32ab0e107c6aa269bfed5632ed18a30af WatchSource:0}: Error finding container 7620b71347287a0eaae4186064a3b4f32ab0e107c6aa269bfed5632ed18a30af: Status 404 returned error can't find the container with id 7620b71347287a0eaae4186064a3b4f32ab0e107c6aa269bfed5632ed18a30af Sep 07 10:17:27.409953 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:27.409848 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" event={"ID":"c684cdc5-ab92-4236-b1d1-6df043a2f531","Type":"ContainerStarted","Data":"15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3"} Sep 07 10:17:27.411885 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:27.411858 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" event={"ID":"401435cb-b468-417d-b7b5-d98787fa98f0","Type":"ContainerStarted","Data":"9664a85867768f7fc2e14c0cc26ce7a0571f2bea1ffeb77a53088356ddeed436"} Sep 07 10:17:27.413217 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:27.413193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" event={"ID":"07a4616c-f300-418e-8027-6b8efc43c741","Type":"ContainerStarted","Data":"7620b71347287a0eaae4186064a3b4f32ab0e107c6aa269bfed5632ed18a30af"} Sep 07 10:17:27.429720 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:27.429669 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" podStartSLOduration=1.3846652210000001 podStartE2EDuration="4.429645307s" totalImagesPullingTime="3.044980086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:27.42767897 +0000 UTC m=+369.413959315" watchObservedRunningTime="2026-09-07 10:17:27.429645307 +0000 UTC m=+369.415925640" Sep 07 10:17:27.448100 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:27.448040 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" podStartSLOduration=1.725827673 podStartE2EDuration="4.448017199s" totalImagesPullingTime="2.722189526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:27.444865663 +0000 UTC m=+369.431145998" watchObservedRunningTime="2026-09-07 10:17:27.448017199 +0000 UTC m=+369.434297537" Sep 07 10:17:27.627823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:27.627796 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:27.962575 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:27.962541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:28.630062 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:28.630028 2565 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Sep 07 10:17:28.630666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:28.630109 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" podUID="c684cdc5-ab92-4236-b1d1-6df043a2f531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Sep 07 10:17:28.968367 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:28.968280 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:29.425468 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:29.425436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:29.426463 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:29.426442 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv" Sep 07 10:17:29.477285 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:29.477193 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88"] Sep 07 10:17:29.628337 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:29.628303 2565 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Sep 07 10:17:29.628529 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:29.628371 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" podUID="c684cdc5-ab92-4236-b1d1-6df043a2f531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Sep 07 10:17:30.431825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:30.431780 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" event={"ID":"07a4616c-f300-418e-8027-6b8efc43c741","Type":"ContainerStarted","Data":"53ee42fd3fc5ca039c4e10d46f658d3fbe2daaa292bb3611c5f8f7042fb2d393"} Sep 07 10:17:30.432291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:30.432095 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:30.432291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:30.432111 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" podUID="c684cdc5-ab92-4236-b1d1-6df043a2f531" containerName="istio-proxy" containerID="cri-o://15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3" gracePeriod=30 Sep 07 10:17:30.449219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:30.449135 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" podStartSLOduration=1.8827216340000001 podStartE2EDuration="4.449113181s" totalImagesPullingTime="2.566391547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:30.44699632 +0000 UTC m=+372.433276654" watchObservedRunningTime="2026-09-07 10:17:30.449113181 +0000 UTC m=+372.435393515" Sep 07 10:17:33.710087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.710051 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh"] Sep 07 10:17:33.713835 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.713816 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.716554 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.716534 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 07 10:17:33.716799 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.716783 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9x86w\"" Sep 07 10:17:33.717026 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.717005 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 07 10:17:33.717137 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.717023 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 07 10:17:33.732755 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.732719 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh"] Sep 07 10:17:33.850580 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.850542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.850580 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.850583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.850837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.850611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.850837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.850686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.850837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.850714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54653ac5-5240-4e23-b341-8d14a8890ca6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.850837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.850785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwq8x\" (UniqueName: \"kubernetes.io/projected/54653ac5-5240-4e23-b341-8d14a8890ca6-kube-api-access-dwq8x\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.850837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.850833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.951933 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.951900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwq8x\" (UniqueName: \"kubernetes.io/projected/54653ac5-5240-4e23-b341-8d14a8890ca6-kube-api-access-dwq8x\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.952093 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.951938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.952093 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.951964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.952093 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.951982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.952267 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.952108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.952267 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.952245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.952373 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.952299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54653ac5-5240-4e23-b341-8d14a8890ca6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.952776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.952730 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.954519 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.954487 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.954619 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.954537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.954619 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.954599 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.954731 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.954710 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54653ac5-5240-4e23-b341-8d14a8890ca6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.966920 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.966866 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwq8x\" (UniqueName: \"kubernetes.io/projected/54653ac5-5240-4e23-b341-8d14a8890ca6-kube-api-access-dwq8x\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:33.967113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:33.967091 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54653ac5-5240-4e23-b341-8d14a8890ca6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-x7qjh\" (UID: \"54653ac5-5240-4e23-b341-8d14a8890ca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:34.024370 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:34.024336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:34.305237 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:34.305214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh"] Sep 07 10:17:34.307202 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:17:34.307175 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54653ac5_5240_4e23_b341_8d14a8890ca6.slice/crio-c58886b869303cc086536f76dfec10f6248e6a335c9fa13172a97f63d0ab9c25 WatchSource:0}: Error finding container c58886b869303cc086536f76dfec10f6248e6a335c9fa13172a97f63d0ab9c25: Status 404 returned error can't find the container with id c58886b869303cc086536f76dfec10f6248e6a335c9fa13172a97f63d0ab9c25 Sep 07 10:17:34.450991 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:34.450960 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" event={"ID":"54653ac5-5240-4e23-b341-8d14a8890ca6","Type":"ContainerStarted","Data":"c58886b869303cc086536f76dfec10f6248e6a335c9fa13172a97f63d0ab9c25"} Sep 07 10:17:35.745894 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.745863 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:35.871321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871287 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-data\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.871321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871322 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-credential-socket\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.871582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-podinfo\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.871582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871373 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-certs\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.871582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871413 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-token\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.871582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871490 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-envoy\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.871582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871519 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzsbn\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-kube-api-access-jzsbn\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.871582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-socket\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.871582 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.871581 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c684cdc5-ab92-4236-b1d1-6df043a2f531-istiod-ca-cert\") pod \"c684cdc5-ab92-4236-b1d1-6df043a2f531\" (UID: \"c684cdc5-ab92-4236-b1d1-6df043a2f531\") " Sep 07 10:17:35.872312 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.872038 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:17:35.872312 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.872233 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:17:35.872828 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.872800 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c684cdc5-ab92-4236-b1d1-6df043a2f531-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:17:35.872901 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.872872 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:17:35.873032 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.873013 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-data" (OuterVolumeSpecName: "istio-data") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:17:35.874878 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.874844 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:17:35.875048 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.875023 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:17:35.876099 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.876068 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-token" (OuterVolumeSpecName: "istio-token") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:17:35.876358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.876324 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-kube-api-access-jzsbn" (OuterVolumeSpecName: "kube-api-access-jzsbn") pod "c684cdc5-ab92-4236-b1d1-6df043a2f531" (UID: "c684cdc5-ab92-4236-b1d1-6df043a2f531"). InnerVolumeSpecName "kube-api-access-jzsbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:17:35.973057 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973002 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzsbn\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-kube-api-access-jzsbn\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:35.973057 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973041 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:35.973057 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973057 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c684cdc5-ab92-4236-b1d1-6df043a2f531-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:35.973347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973069 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:35.973347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973080 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:35.973347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973091 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:35.973347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973101 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:35.973347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973112 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:35.973347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:35.973122 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c684cdc5-ab92-4236-b1d1-6df043a2f531-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:17:36.469916 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.466542 2565 generic.go:356] "Generic (PLEG): container finished" podID="c684cdc5-ab92-4236-b1d1-6df043a2f531" containerID="15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3" exitCode=0 Sep 07 10:17:36.469916 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.466763 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" event={"ID":"c684cdc5-ab92-4236-b1d1-6df043a2f531","Type":"ContainerDied","Data":"15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3"} Sep 07 10:17:36.469916 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.466836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" event={"ID":"c684cdc5-ab92-4236-b1d1-6df043a2f531","Type":"ContainerDied","Data":"f03e3596438f56bbeac2e1bcec91b8d37ad341abd97101e94948a851628c4782"} Sep 07 10:17:36.469916 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.466869 2565 scope.go:117] "RemoveContainer" containerID="15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3" Sep 07 10:17:36.469916 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.467108 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88" Sep 07 10:17:36.516024 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.515986 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88"] Sep 07 10:17:36.528613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.528577 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-28v88"] Sep 07 10:17:36.546515 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.546476 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c684cdc5-ab92-4236-b1d1-6df043a2f531" path="/var/lib/kubelet/pods/c684cdc5-ab92-4236-b1d1-6df043a2f531/volumes" Sep 07 10:17:36.909909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.909755 2565 scope.go:117] "RemoveContainer" containerID="15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3" Sep 07 10:17:36.910231 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:17:36.910095 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3\": container with ID starting with 15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3 not found: ID does not exist" containerID="15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3" Sep 07 10:17:36.910231 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.910121 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3"} err="failed to get container status \"15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3\": rpc error: code = NotFound desc = could not find container \"15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3\": container with ID starting with 15032a80d00cf191287fd39bcca1d9581c7bd0b49b32c45ead9e6b492be5a1a3 not found: ID does not exist" Sep 07 10:17:36.969953 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.969916 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:17:36.970053 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:36.969979 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:17:37.474670 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:37.474635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" event={"ID":"54653ac5-5240-4e23-b341-8d14a8890ca6","Type":"ContainerStarted","Data":"53d514d5cef62a0dd9963c44ede6f48d7da7809f24b57e96bb45c7be82d5a05b"} Sep 07 10:17:37.475138 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:37.475114 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:37.501898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:37.501835 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" podStartSLOduration=1.841516521 podStartE2EDuration="4.50181267s" totalImagesPullingTime="2.660296149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:37.499394025 +0000 UTC m=+379.485674362" watchObservedRunningTime="2026-09-07 10:17:37.50181267 +0000 UTC m=+379.488093005" Sep 07 10:17:38.482473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:38.482446 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x7qjh" Sep 07 10:17:41.439297 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:41.439268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zhpzs" Sep 07 10:17:52.653113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:17:52.653075 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbfc8d6f6-z4t2q"] Sep 07 10:18:17.679796 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:17.679734 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fbfc8d6f6-z4t2q" podUID="bc198495-2f01-449a-a864-f097584b4fb9" containerName="console" containerID="cri-o://161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c" gracePeriod=15 Sep 07 10:18:17.927071 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:17.927046 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:18:18.049413 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.049374 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-oauth-serving-cert\") pod \"bc198495-2f01-449a-a864-f097584b4fb9\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " Sep 07 10:18:18.049413 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.049424 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-serving-cert\") pod \"bc198495-2f01-449a-a864-f097584b4fb9\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " Sep 07 10:18:18.049670 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.049449 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnwhk\" (UniqueName: \"kubernetes.io/projected/bc198495-2f01-449a-a864-f097584b4fb9-kube-api-access-fnwhk\") pod \"bc198495-2f01-449a-a864-f097584b4fb9\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " Sep 07 10:18:18.049670 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.049471 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-console-config\") pod \"bc198495-2f01-449a-a864-f097584b4fb9\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " Sep 07 10:18:18.049670 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.049507 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-trusted-ca-bundle\") pod \"bc198495-2f01-449a-a864-f097584b4fb9\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " Sep 07 10:18:18.049670 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.049537 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-oauth-config\") pod \"bc198495-2f01-449a-a864-f097584b4fb9\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " Sep 07 10:18:18.049876 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.049714 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-service-ca\") pod \"bc198495-2f01-449a-a864-f097584b4fb9\" (UID: \"bc198495-2f01-449a-a864-f097584b4fb9\") " Sep 07 10:18:18.049876 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.049828 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bc198495-2f01-449a-a864-f097584b4fb9" (UID: "bc198495-2f01-449a-a864-f097584b4fb9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:18:18.050116 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.050070 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bc198495-2f01-449a-a864-f097584b4fb9" (UID: "bc198495-2f01-449a-a864-f097584b4fb9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:18:18.050116 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.050091 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-console-config" (OuterVolumeSpecName: "console-config") pod "bc198495-2f01-449a-a864-f097584b4fb9" (UID: "bc198495-2f01-449a-a864-f097584b4fb9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:18:18.050509 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.050137 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-service-ca" (OuterVolumeSpecName: "service-ca") pod "bc198495-2f01-449a-a864-f097584b4fb9" (UID: "bc198495-2f01-449a-a864-f097584b4fb9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:18:18.050509 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.050221 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-oauth-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:18:18.051796 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.051769 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc198495-2f01-449a-a864-f097584b4fb9-kube-api-access-fnwhk" (OuterVolumeSpecName: "kube-api-access-fnwhk") pod "bc198495-2f01-449a-a864-f097584b4fb9" (UID: "bc198495-2f01-449a-a864-f097584b4fb9"). InnerVolumeSpecName "kube-api-access-fnwhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:18:18.051910 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.051843 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bc198495-2f01-449a-a864-f097584b4fb9" (UID: "bc198495-2f01-449a-a864-f097584b4fb9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:18:18.051910 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.051892 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bc198495-2f01-449a-a864-f097584b4fb9" (UID: "bc198495-2f01-449a-a864-f097584b4fb9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:18:18.151745 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.151708 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:18:18.151745 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.151738 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnwhk\" (UniqueName: \"kubernetes.io/projected/bc198495-2f01-449a-a864-f097584b4fb9-kube-api-access-fnwhk\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:18:18.151745 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.151751 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-console-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:18:18.151988 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.151764 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-trusted-ca-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:18:18.151988 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.151775 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc198495-2f01-449a-a864-f097584b4fb9-console-oauth-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:18:18.151988 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.151787 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc198495-2f01-449a-a864-f097584b4fb9-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:18:18.675260 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.675227 2565 generic.go:356] "Generic (PLEG): container finished" podID="bc198495-2f01-449a-a864-f097584b4fb9" containerID="161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c" exitCode=0 Sep 07 10:18:18.675422 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.675268 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbfc8d6f6-z4t2q" Sep 07 10:18:18.675422 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.675315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbfc8d6f6-z4t2q" event={"ID":"bc198495-2f01-449a-a864-f097584b4fb9","Type":"ContainerDied","Data":"161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c"} Sep 07 10:18:18.675422 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.675357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbfc8d6f6-z4t2q" event={"ID":"bc198495-2f01-449a-a864-f097584b4fb9","Type":"ContainerDied","Data":"a736c9292c02cfd5f0eaed1b06357540bfebbb0e9d13699de979b1d3b8cacc50"} Sep 07 10:18:18.675422 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.675377 2565 scope.go:117] "RemoveContainer" containerID="161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c" Sep 07 10:18:18.684243 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.684085 2565 scope.go:117] "RemoveContainer" containerID="161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c" Sep 07 10:18:18.684457 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:18:18.684407 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c\": container with ID starting with 161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c not found: ID does not exist" containerID="161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c" Sep 07 10:18:18.684457 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.684441 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c"} err="failed to get container status \"161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c\": rpc error: code = NotFound desc = could not find container \"161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c\": container with ID starting with 161de83842b8ecfeb0262c287b038a502afeecfeb63be6baa1292701978a631c not found: ID does not exist" Sep 07 10:18:18.690896 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.690871 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbfc8d6f6-z4t2q"] Sep 07 10:18:18.694915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:18.694892 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fbfc8d6f6-z4t2q"] Sep 07 10:18:20.543611 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:20.543574 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc198495-2f01-449a-a864-f097584b4fb9" path="/var/lib/kubelet/pods/bc198495-2f01-449a-a864-f097584b4fb9/volumes" Sep 07 10:18:43.793823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:43.793734 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-vtq76" podUID="ca012ec4-4ff9-41e3-9135-3f7b477068d4" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:18:43.794353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:43.793864 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:18:43.794496 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:18:43.794475 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-vtq76" podUID="ca012ec4-4ff9-41e3-9135-3f7b477068d4" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:19:13.795723 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:13.795691 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vtq76" Sep 07 10:19:38.524208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.524174 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f"] Sep 07 10:19:38.524733 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.524701 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc198495-2f01-449a-a864-f097584b4fb9" containerName="console" Sep 07 10:19:38.524816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.524736 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc198495-2f01-449a-a864-f097584b4fb9" containerName="console" Sep 07 10:19:38.524816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.524799 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c684cdc5-ab92-4236-b1d1-6df043a2f531" containerName="istio-proxy" Sep 07 10:19:38.524816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.524809 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c684cdc5-ab92-4236-b1d1-6df043a2f531" containerName="istio-proxy" Sep 07 10:19:38.524985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.524895 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c684cdc5-ab92-4236-b1d1-6df043a2f531" containerName="istio-proxy" Sep 07 10:19:38.524985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.524907 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc198495-2f01-449a-a864-f097584b4fb9" containerName="console" Sep 07 10:19:38.529122 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.529102 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.531796 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.531771 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7hx6t\"" Sep 07 10:19:38.544322 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.544301 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f"] Sep 07 10:19:38.665404 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkl5\" (UniqueName: \"kubernetes.io/projected/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-kube-api-access-rjkl5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.665404 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.665600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665446 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.665600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.665600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.665600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665535 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.665600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665565 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.665600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.665795 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.665611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.767150 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.767362 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.767362 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.767362 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.767362 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767313 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.767362 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.767673 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.767857 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.768016 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkl5\" (UniqueName: \"kubernetes.io/projected/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-kube-api-access-rjkl5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.768016 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.767988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.768200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.768011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.768200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.768146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.768200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.768188 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.768755 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.768736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.770191 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.770152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.770445 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.770427 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.776079 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.776019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.776212 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.776194 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkl5\" (UniqueName: \"kubernetes.io/projected/114f0b81-1d6c-45c7-a59f-0d5d210c49b2-kube-api-access-rjkl5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-94w8f\" (UID: \"114f0b81-1d6c-45c7-a59f-0d5d210c49b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:38.842044 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:38.842011 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:39.119213 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:39.119143 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f"] Sep 07 10:19:39.121987 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:19:39.121959 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114f0b81_1d6c_45c7_a59f_0d5d210c49b2.slice/crio-c2e6ac55b81ce259feba7f0aea67a6bfcf08dc76acf02cc0facb2a70cb0ac9c2 WatchSource:0}: Error finding container c2e6ac55b81ce259feba7f0aea67a6bfcf08dc76acf02cc0facb2a70cb0ac9c2: Status 404 returned error can't find the container with id c2e6ac55b81ce259feba7f0aea67a6bfcf08dc76acf02cc0facb2a70cb0ac9c2 Sep 07 10:19:39.124210 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:39.124174 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:19:39.124264 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:39.124252 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:19:39.124299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:39.124283 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:19:40.069912 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:40.069874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" event={"ID":"114f0b81-1d6c-45c7-a59f-0d5d210c49b2","Type":"ContainerStarted","Data":"2e1564fdfa88b003e4e9f3163e923f524a06a449a27782185a9c4bed461a1833"} Sep 07 10:19:40.070379 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:40.069955 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" event={"ID":"114f0b81-1d6c-45c7-a59f-0d5d210c49b2","Type":"ContainerStarted","Data":"c2e6ac55b81ce259feba7f0aea67a6bfcf08dc76acf02cc0facb2a70cb0ac9c2"} Sep 07 10:19:40.089629 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:40.089579 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" podStartSLOduration=2.089563401 podStartE2EDuration="2.089563401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:40.088097895 +0000 UTC m=+502.074378229" watchObservedRunningTime="2026-09-07 10:19:40.089563401 +0000 UTC m=+502.075843735" Sep 07 10:19:40.842146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:40.842110 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:40.847540 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:40.847518 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:41.074795 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:41.074758 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:19:41.075686 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:19:41.075668 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:21:12.373207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.373142 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jxjws"] Sep 07 10:21:12.376531 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.376514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:12.378946 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.378920 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mpqn5\"" Sep 07 10:21:12.379066 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.378974 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 07 10:21:12.387031 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.387008 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jxjws"] Sep 07 10:21:12.510993 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.510953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzsqb\" (UniqueName: \"kubernetes.io/projected/170e144a-1758-4aa1-9f7f-9130f65ce97a-kube-api-access-wzsqb\") pod \"payload-processing-5b54b7fbb7-jxjws\" (UID: \"170e144a-1758-4aa1-9f7f-9130f65ce97a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:12.511198 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.510999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170e144a-1758-4aa1-9f7f-9130f65ce97a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jxjws\" (UID: \"170e144a-1758-4aa1-9f7f-9130f65ce97a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:12.611865 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.611827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzsqb\" (UniqueName: \"kubernetes.io/projected/170e144a-1758-4aa1-9f7f-9130f65ce97a-kube-api-access-wzsqb\") pod \"payload-processing-5b54b7fbb7-jxjws\" (UID: \"170e144a-1758-4aa1-9f7f-9130f65ce97a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:12.611865 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.611868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170e144a-1758-4aa1-9f7f-9130f65ce97a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jxjws\" (UID: \"170e144a-1758-4aa1-9f7f-9130f65ce97a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:12.612572 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.612553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/170e144a-1758-4aa1-9f7f-9130f65ce97a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jxjws\" (UID: \"170e144a-1758-4aa1-9f7f-9130f65ce97a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:12.620216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.620188 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzsqb\" (UniqueName: \"kubernetes.io/projected/170e144a-1758-4aa1-9f7f-9130f65ce97a-kube-api-access-wzsqb\") pod \"payload-processing-5b54b7fbb7-jxjws\" (UID: \"170e144a-1758-4aa1-9f7f-9130f65ce97a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:12.688321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.688232 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:12.961210 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:12.961186 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jxjws"] Sep 07 10:21:12.964094 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:21:12.964059 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170e144a_1758_4aa1_9f7f_9130f65ce97a.slice/crio-b79d5e84f01dd8fd552178b1dfb56bfd8052018374df9a635d14c65baba34192 WatchSource:0}: Error finding container b79d5e84f01dd8fd552178b1dfb56bfd8052018374df9a635d14c65baba34192: Status 404 returned error can't find the container with id b79d5e84f01dd8fd552178b1dfb56bfd8052018374df9a635d14c65baba34192 Sep 07 10:21:13.522772 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:13.522729 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" event={"ID":"170e144a-1758-4aa1-9f7f-9130f65ce97a","Type":"ContainerStarted","Data":"b79d5e84f01dd8fd552178b1dfb56bfd8052018374df9a635d14c65baba34192"} Sep 07 10:21:15.534962 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:15.534921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" event={"ID":"170e144a-1758-4aa1-9f7f-9130f65ce97a","Type":"ContainerStarted","Data":"6a4603bc9475adf38e97a8d9ac6e2a094149113755fd3adf97a66dd1ef94e103"} Sep 07 10:21:15.535521 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:15.535046 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:15.555641 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:15.555570 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" podStartSLOduration=1.693892576 podStartE2EDuration="3.555547813s" totalImagesPullingTime="1.861655237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:15.552071212 +0000 UTC m=+597.538351549" watchObservedRunningTime="2026-09-07 10:21:15.555547813 +0000 UTC m=+597.541828152" Sep 07 10:21:18.484423 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:18.484395 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:21:18.485018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:18.485003 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:21:22.585389 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:22.585353 2565 generic.go:356] "Generic (PLEG): container finished" podID="114f0b81-1d6c-45c7-a59f-0d5d210c49b2" containerID="2e1564fdfa88b003e4e9f3163e923f524a06a449a27782185a9c4bed461a1833" exitCode=0 Sep 07 10:21:22.585857 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:22.585427 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" event={"ID":"114f0b81-1d6c-45c7-a59f-0d5d210c49b2","Type":"ContainerDied","Data":"2e1564fdfa88b003e4e9f3163e923f524a06a449a27782185a9c4bed461a1833"} Sep 07 10:21:22.585917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:22.585870 2565 scope.go:117] "RemoveContainer" containerID="2e1564fdfa88b003e4e9f3163e923f524a06a449a27782185a9c4bed461a1833" Sep 07 10:21:22.587018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:22.586985 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:21:22.587139 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:22.587069 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:21:22.587139 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:22.587120 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:21:23.592623 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:23.592589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" event={"ID":"114f0b81-1d6c-45c7-a59f-0d5d210c49b2","Type":"ContainerStarted","Data":"491378ca544d539fe46e8b2f7877c318a64ec37b2a8abdf28006b94a33f2b690"} Sep 07 10:21:23.843321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:23.843227 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:21:23.844592 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:23.844562 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-94w8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:21:23.844832 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:23.844798 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" podUID="114f0b81-1d6c-45c7-a59f-0d5d210c49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 10:21:24.843078 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:24.843039 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-94w8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:21:24.843711 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:24.843677 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" podUID="114f0b81-1d6c-45c7-a59f-0d5d210c49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 10:21:25.843341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:25.843303 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-94w8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:21:25.844423 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:25.844384 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" podUID="114f0b81-1d6c-45c7-a59f-0d5d210c49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 10:21:26.075187 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:26.075125 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:21:26.547357 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:26.547330 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jxjws" Sep 07 10:21:26.842660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:26.842558 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-94w8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:21:26.842660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:26.842629 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" podUID="114f0b81-1d6c-45c7-a59f-0d5d210c49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 10:21:27.843504 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:27.843462 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-94w8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:21:27.844206 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:27.844144 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" podUID="114f0b81-1d6c-45c7-a59f-0d5d210c49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 07 10:21:28.846189 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:28.846142 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:21:28.847270 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:28.847247 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-94w8f" Sep 07 10:21:43.392926 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.392838 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7"] Sep 07 10:21:43.396411 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.396396 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.400055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.400024 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 07 10:21:43.400203 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.400090 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 07 10:21:43.400203 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.400091 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 07 10:21:43.400203 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.400142 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-87q54\"" Sep 07 10:21:43.405331 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.405308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7"] Sep 07 10:21:43.525938 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.525902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a008c9dc-6f53-4f35-9cab-10cb26583986-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.526112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.525962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.526112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.526048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.526112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.526097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.526251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.526154 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hpl\" (UniqueName: \"kubernetes.io/projected/a008c9dc-6f53-4f35-9cab-10cb26583986-kube-api-access-k4hpl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.526251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.526210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.626681 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.626644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.626870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.626709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.626870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.626743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hpl\" (UniqueName: \"kubernetes.io/projected/a008c9dc-6f53-4f35-9cab-10cb26583986-kube-api-access-k4hpl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.626870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.626769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.626870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.626801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a008c9dc-6f53-4f35-9cab-10cb26583986-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.626870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.626865 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.627131 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.627096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.627131 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.627112 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.627286 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.627253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.629266 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.629234 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a008c9dc-6f53-4f35-9cab-10cb26583986-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.629441 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.629426 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a008c9dc-6f53-4f35-9cab-10cb26583986-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.634632 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.634609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hpl\" (UniqueName: \"kubernetes.io/projected/a008c9dc-6f53-4f35-9cab-10cb26583986-kube-api-access-k4hpl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7\" (UID: \"a008c9dc-6f53-4f35-9cab-10cb26583986\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.706832 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.706751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:43.988784 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.988706 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7"] Sep 07 10:21:43.991023 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:21:43.990996 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda008c9dc_6f53_4f35_9cab_10cb26583986.slice/crio-313430ba57e7bb446ff29229174f0e0c26a3228464dd0b13fd37dbb6571059ee WatchSource:0}: Error finding container 313430ba57e7bb446ff29229174f0e0c26a3228464dd0b13fd37dbb6571059ee: Status 404 returned error can't find the container with id 313430ba57e7bb446ff29229174f0e0c26a3228464dd0b13fd37dbb6571059ee Sep 07 10:21:43.992477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:43.992457 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:21:44.703503 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:44.703464 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" event={"ID":"a008c9dc-6f53-4f35-9cab-10cb26583986","Type":"ContainerStarted","Data":"313430ba57e7bb446ff29229174f0e0c26a3228464dd0b13fd37dbb6571059ee"} Sep 07 10:21:45.710435 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:45.710401 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" event={"ID":"a008c9dc-6f53-4f35-9cab-10cb26583986","Type":"ContainerStarted","Data":"258925f51d9c8048dbe7032eaa692e568e3622be9cffb8f70feb2bac02e47e64"} Sep 07 10:21:45.727539 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:45.727487 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" podStartSLOduration=1.5990063970000001 podStartE2EDuration="2.72747143s" totalImagesPullingTime="1.128465033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:45.725497122 +0000 UTC m=+627.711777450" watchObservedRunningTime="2026-09-07 10:21:45.72747143 +0000 UTC m=+627.713751764" Sep 07 10:21:53.707270 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:53.707221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:53.707750 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:53.707314 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:53.719801 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:53.719774 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:21:53.762096 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:21:53.762061 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7" Sep 07 10:22:04.996647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:04.996605 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2"] Sep 07 10:22:05.001072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.001050 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.005613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.005593 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 07 10:22:05.013529 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.013505 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2"] Sep 07 10:22:05.027894 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.027863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9840e152-cec6-442a-ba65-3c10d6bfe128-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.028045 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.027936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.028045 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.027994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.028146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.028126 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.028228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.028186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzt7\" (UniqueName: \"kubernetes.io/projected/9840e152-cec6-442a-ba65-3c10d6bfe128-kube-api-access-7pzt7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.028283 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.028233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.129479 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.129685 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.129685 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.129685 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzt7\" (UniqueName: \"kubernetes.io/projected/9840e152-cec6-442a-ba65-3c10d6bfe128-kube-api-access-7pzt7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.129685 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.129900 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9840e152-cec6-442a-ba65-3c10d6bfe128-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.129955 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129927 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.130010 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.130010 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.129992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.131957 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.131927 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9840e152-cec6-442a-ba65-3c10d6bfe128-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.132442 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.132423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9840e152-cec6-442a-ba65-3c10d6bfe128-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.140410 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.140382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzt7\" (UniqueName: \"kubernetes.io/projected/9840e152-cec6-442a-ba65-3c10d6bfe128-kube-api-access-7pzt7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b59c2\" (UID: \"9840e152-cec6-442a-ba65-3c10d6bfe128\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.311832 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.311741 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:05.584336 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.584309 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2"] Sep 07 10:22:05.586442 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:22:05.586414 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9840e152_cec6_442a_ba65_3c10d6bfe128.slice/crio-10cd26149ae3859c83588c57326717d9a1161964ab20cf13ea28fc9545fe0185 WatchSource:0}: Error finding container 10cd26149ae3859c83588c57326717d9a1161964ab20cf13ea28fc9545fe0185: Status 404 returned error can't find the container with id 10cd26149ae3859c83588c57326717d9a1161964ab20cf13ea28fc9545fe0185 Sep 07 10:22:05.811824 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:05.811794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" event={"ID":"9840e152-cec6-442a-ba65-3c10d6bfe128","Type":"ContainerStarted","Data":"10cd26149ae3859c83588c57326717d9a1161964ab20cf13ea28fc9545fe0185"} Sep 07 10:22:06.818689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:06.818658 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" event={"ID":"9840e152-cec6-442a-ba65-3c10d6bfe128","Type":"ContainerStarted","Data":"d3fce6e2934d1f1a31136dd5ea1a08ad9d0aceea9b0122c081612bb50cb9239d"} Sep 07 10:22:06.835128 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:06.835064 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" podStartSLOduration=2.644816728 podStartE2EDuration="2.835042781s" totalImagesPullingTime="190.226053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:22:06.834478 +0000 UTC m=+648.820758334" watchObservedRunningTime="2026-09-07 10:22:06.835042781 +0000 UTC m=+648.821323116" Sep 07 10:22:15.312926 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:15.312863 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:15.312926 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:15.312937 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:15.326009 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:15.325980 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:22:15.876295 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:22:15.876264 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b59c2" Sep 07 10:24:15.726337 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.726302 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml"] Sep 07 10:24:15.731137 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.730449 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.733216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.733193 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-05ecbf-openshift-default-dockercfg-xftqw\"" Sep 07 10:24:15.741974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.741951 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml"] Sep 07 10:24:15.824364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42k8\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-kube-api-access-m42k8\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.824520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-envoy\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.824520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-istio-token\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.824520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-socket\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.824520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639c2568-bb85-49cd-9a30-e228190a0acf-istiod-ca-cert\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.824676 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-credential-socket\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.824676 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824571 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639c2568-bb85-49cd-9a30-e228190a0acf-istio-podinfo\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.824676 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-certs\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.824676 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.824620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-data\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.925612 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-credential-socket\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.925612 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639c2568-bb85-49cd-9a30-e228190a0acf-istio-podinfo\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.925861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-certs\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.925861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925752 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-data\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.925861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42k8\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-kube-api-access-m42k8\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.925861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-envoy\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.926087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-istio-token\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.926087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-socket\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.926087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639c2568-bb85-49cd-9a30-e228190a0acf-istiod-ca-cert\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.926087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.925965 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-certs\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.926087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.926007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-credential-socket\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.926087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.926058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-data\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.926438 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.926420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-socket\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.926614 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.926596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639c2568-bb85-49cd-9a30-e228190a0acf-istiod-ca-cert\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.928329 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.928308 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-envoy\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.928438 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.928335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639c2568-bb85-49cd-9a30-e228190a0acf-istio-podinfo\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.934446 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.934429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-istio-token\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:15.934716 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:15.934697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42k8\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-kube-api-access-m42k8\") pod \"e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:16.043634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:16.043562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:16.329407 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:16.329373 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml"] Sep 07 10:24:16.332256 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:24:16.332230 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639c2568_bb85_49cd_9a30_e228190a0acf.slice/crio-704f1bce154bfbf1c3b4ef838c2e88fb0088b64f42dbf60ba97ebd296425177a WatchSource:0}: Error finding container 704f1bce154bfbf1c3b4ef838c2e88fb0088b64f42dbf60ba97ebd296425177a: Status 404 returned error can't find the container with id 704f1bce154bfbf1c3b4ef838c2e88fb0088b64f42dbf60ba97ebd296425177a Sep 07 10:24:16.334509 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:16.334473 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:16.334590 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:16.334546 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:16.334590 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:16.334576 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:16.466540 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:16.466506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" event={"ID":"639c2568-bb85-49cd-9a30-e228190a0acf","Type":"ContainerStarted","Data":"9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254"} Sep 07 10:24:16.466735 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:16.466574 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" event={"ID":"639c2568-bb85-49cd-9a30-e228190a0acf","Type":"ContainerStarted","Data":"704f1bce154bfbf1c3b4ef838c2e88fb0088b64f42dbf60ba97ebd296425177a"} Sep 07 10:24:16.485472 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:16.485420 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" podStartSLOduration=1.485402787 podStartE2EDuration="1.485402787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:16.483302129 +0000 UTC m=+778.469582465" watchObservedRunningTime="2026-09-07 10:24:16.485402787 +0000 UTC m=+778.471683122" Sep 07 10:24:17.044712 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:17.044678 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:17.049272 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:17.049249 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:17.471234 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:17.471208 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:17.472102 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:17.472081 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:23.425532 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.425496 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l"] Sep 07 10:24:23.429498 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.429476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.431978 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.431951 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8ec6b6b5-openshift-default-dockercfg-f77mt\"" Sep 07 10:24:23.443302 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.443280 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l"] Sep 07 10:24:23.503444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-token\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.503617 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-certs\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.503617 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-credential-socket\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.503617 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503501 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-envoy\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.503617 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-data\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.503617 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istiod-ca-cert\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.503780 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54gt\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-kube-api-access-z54gt\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.503780 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-podinfo\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.503780 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.503731 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-socket\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.604785 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.604751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-data\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.604785 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.604784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istiod-ca-cert\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605016 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.604835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54gt\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-kube-api-access-z54gt\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605016 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.604861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-podinfo\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605016 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.604991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-socket\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605205 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-token\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605205 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605132 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-certs\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-credential-socket\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-envoy\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-socket\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-credential-socket\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-data\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605981 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istiod-ca-cert\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.605981 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.605973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-certs\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.607549 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.607528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-envoy\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.607647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.607556 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-podinfo\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.616405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.616385 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-token\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.616497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.616425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54gt\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-kube-api-access-z54gt\") pod \"e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:23.741429 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:23.741391 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:24.019668 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:24.019642 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l"] Sep 07 10:24:24.021678 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:24:24.021648 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426da8e4_5ecc_4da2_89ef_b695d8cc17ce.slice/crio-b0686cc31fd5372906c9d4c9418639116dc1d11e27fd793ac71753f8ff94ed1e WatchSource:0}: Error finding container b0686cc31fd5372906c9d4c9418639116dc1d11e27fd793ac71753f8ff94ed1e: Status 404 returned error can't find the container with id b0686cc31fd5372906c9d4c9418639116dc1d11e27fd793ac71753f8ff94ed1e Sep 07 10:24:24.023779 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:24.023744 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:24.023860 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:24.023811 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:24.023860 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:24.023835 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:24.509839 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:24.509802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" event={"ID":"426da8e4-5ecc-4da2-89ef-b695d8cc17ce","Type":"ContainerStarted","Data":"b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131"} Sep 07 10:24:24.510247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:24.509886 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" event={"ID":"426da8e4-5ecc-4da2-89ef-b695d8cc17ce","Type":"ContainerStarted","Data":"b0686cc31fd5372906c9d4c9418639116dc1d11e27fd793ac71753f8ff94ed1e"} Sep 07 10:24:24.529779 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:24.529721 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podStartSLOduration=1.529706488 podStartE2EDuration="1.529706488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:24.527970404 +0000 UTC m=+786.514250739" watchObservedRunningTime="2026-09-07 10:24:24.529706488 +0000 UTC m=+786.515986822" Sep 07 10:24:24.741857 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:24.741823 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:25.182499 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.182463 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf"] Sep 07 10:24:25.186137 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.186117 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.188853 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.188832 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ff5470-openshift-default-dockercfg-6j9jm\"" Sep 07 10:24:25.201887 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.201864 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf"] Sep 07 10:24:25.222204 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4576c52d-4883-44c1-b5a4-478251329113-istiod-ca-cert\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.222322 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-envoy\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.222322 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-data\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.222396 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-credential-socket\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.222437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-socket\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.222437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222423 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtmx\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-kube-api-access-smtmx\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.222516 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4576c52d-4883-44c1-b5a4-478251329113-istio-podinfo\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.222562 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-istio-token\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.222596 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.222581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-certs\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-credential-socket\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323563 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-socket\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323563 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtmx\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-kube-api-access-smtmx\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323563 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4576c52d-4883-44c1-b5a4-478251329113-istio-podinfo\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323563 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-istio-token\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323563 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-certs\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4576c52d-4883-44c1-b5a4-478251329113-istiod-ca-cert\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-envoy\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-data\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.323809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323755 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-credential-socket\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.324009 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-socket\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.324009 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.323999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-certs\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.324101 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.324036 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-data\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.324386 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.324365 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4576c52d-4883-44c1-b5a4-478251329113-istiod-ca-cert\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.325985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.325962 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-envoy\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.326265 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.326249 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4576c52d-4883-44c1-b5a4-478251329113-istio-podinfo\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.332029 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.332007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-istio-token\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.332233 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.332216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtmx\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-kube-api-access-smtmx\") pod \"e2e-autores-ff5470-openshift-default-76766d54bb-v77jf\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.497589 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.497553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:25.747434 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.747398 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:25.773470 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.773448 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf"] Sep 07 10:24:25.775025 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:24:25.774998 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4576c52d_4883_44c1_b5a4_478251329113.slice/crio-0f20279308a62770b5d1202f454ec4048279a152ecbfa914c773b9462f2b2524 WatchSource:0}: Error finding container 0f20279308a62770b5d1202f454ec4048279a152ecbfa914c773b9462f2b2524: Status 404 returned error can't find the container with id 0f20279308a62770b5d1202f454ec4048279a152ecbfa914c773b9462f2b2524 Sep 07 10:24:25.777466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.777427 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:25.777542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.777513 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:25.777793 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:25.777564 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:26.525009 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:26.524976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" event={"ID":"4576c52d-4883-44c1-b5a4-478251329113","Type":"ContainerStarted","Data":"ab448cacd7d0f1d70dacd9cc313db0d3cc8d2f81ae8d926dac49d68301f9fd10"} Sep 07 10:24:26.525239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:26.525034 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" event={"ID":"4576c52d-4883-44c1-b5a4-478251329113","Type":"ContainerStarted","Data":"0f20279308a62770b5d1202f454ec4048279a152ecbfa914c773b9462f2b2524"} Sep 07 10:24:26.525239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:26.525225 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:26.526059 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:26.526043 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:26.563511 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:26.563457 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" podStartSLOduration=1.563439376 podStartE2EDuration="1.563439376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:26.560214986 +0000 UTC m=+788.546495319" watchObservedRunningTime="2026-09-07 10:24:26.563439376 +0000 UTC m=+788.549719711" Sep 07 10:24:27.498673 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:27.498642 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:27.503758 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:27.503730 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:27.530521 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:27.530493 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:27.531425 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:27.531403 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:30.260208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.260171 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv"] Sep 07 10:24:30.264086 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.264065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:30.266678 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.266656 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ff5470-dockercfg-j2r76\"" Sep 07 10:24:30.266786 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.266665 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ff5470\"" Sep 07 10:24:30.276973 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.276945 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv"] Sep 07 10:24:30.372814 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.372768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72b2h\" (UniqueName: \"kubernetes.io/projected/d86c3d94-0aba-4947-8540-3e08818173eb-kube-api-access-72b2h\") pod \"payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv\" (UID: \"d86c3d94-0aba-4947-8540-3e08818173eb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:30.373005 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.372847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86c3d94-0aba-4947-8540-3e08818173eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv\" (UID: \"d86c3d94-0aba-4947-8540-3e08818173eb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:30.474320 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.474281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86c3d94-0aba-4947-8540-3e08818173eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv\" (UID: \"d86c3d94-0aba-4947-8540-3e08818173eb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:30.474518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.474451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72b2h\" (UniqueName: \"kubernetes.io/projected/d86c3d94-0aba-4947-8540-3e08818173eb-kube-api-access-72b2h\") pod \"payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv\" (UID: \"d86c3d94-0aba-4947-8540-3e08818173eb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:30.475033 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.475016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86c3d94-0aba-4947-8540-3e08818173eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv\" (UID: \"d86c3d94-0aba-4947-8540-3e08818173eb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:30.482629 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.482607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72b2h\" (UniqueName: \"kubernetes.io/projected/d86c3d94-0aba-4947-8540-3e08818173eb-kube-api-access-72b2h\") pod \"payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv\" (UID: \"d86c3d94-0aba-4947-8540-3e08818173eb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:30.574738 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.574648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:30.858555 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:30.851397 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv"] Sep 07 10:24:31.567721 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:31.567691 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" event={"ID":"d86c3d94-0aba-4947-8540-3e08818173eb","Type":"ContainerStarted","Data":"89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2"} Sep 07 10:24:31.568108 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:31.567738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" event={"ID":"d86c3d94-0aba-4947-8540-3e08818173eb","Type":"ContainerStarted","Data":"c99b102e9e0ed17f857e28a09c6de2e7b055785612ad2acb145977a5617b2fc8"} Sep 07 10:24:31.568108 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:31.567783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:31.586002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:31.585946 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" podStartSLOduration=1.585923239 podStartE2EDuration="1.585923239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:31.581908592 +0000 UTC m=+793.568188925" watchObservedRunningTime="2026-09-07 10:24:31.585923239 +0000 UTC m=+793.572203577" Sep 07 10:24:39.615140 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:39.615062 2565 generic.go:356] "Generic (PLEG): container finished" podID="639c2568-bb85-49cd-9a30-e228190a0acf" containerID="9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254" exitCode=0 Sep 07 10:24:39.615578 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:39.615133 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" event={"ID":"639c2568-bb85-49cd-9a30-e228190a0acf","Type":"ContainerDied","Data":"9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254"} Sep 07 10:24:39.615621 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:39.615600 2565 scope.go:117] "RemoveContainer" containerID="9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254" Sep 07 10:24:39.616690 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:39.616659 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:39.616782 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:39.616732 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:39.616782 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:39.616770 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:40.623462 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:40.623374 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" event={"ID":"639c2568-bb85-49cd-9a30-e228190a0acf","Type":"ContainerStarted","Data":"6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238"} Sep 07 10:24:41.047455 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:41.047424 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:41.047716 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:41.047691 2565 patch_prober.go:28] interesting pod/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:24:41.047812 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:41.047747 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 07 10:24:42.044634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:42.044587 2565 patch_prober.go:28] interesting pod/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:24:42.045181 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:42.044676 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 07 10:24:42.575230 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:42.575193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:24:43.044414 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:43.044368 2565 patch_prober.go:28] interesting pod/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:24:43.044609 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:43.044460 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 07 10:24:43.571457 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:43.571405 2565 patch_prober.go:28] interesting pod/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:24:43.571975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:43.571490 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:24:43.960248 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:43.958641 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht"] Sep 07 10:24:43.965183 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:43.962932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:43.969182 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:43.967561 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f17c1f-openshift-default-dockercfg-fpb7x\"" Sep 07 10:24:44.008041 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.008000 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht"] Sep 07 10:24:44.057756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.057718 2565 patch_prober.go:28] interesting pod/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:24:44.058030 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.057998 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 07 10:24:44.058779 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.058745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-token\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.059010 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.058988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-credential-socket\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.059181 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.059143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59fb\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-kube-api-access-c59fb\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.059315 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.059297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-envoy\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.059427 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.059411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-data\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.059645 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.059627 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-podinfo\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.059772 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.059757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istiod-ca-cert\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.059908 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.059894 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-socket\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.060044 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.060031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-certs\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.161414 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-socket\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.161414 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-certs\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.161713 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-token\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.161713 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-credential-socket\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.161713 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59fb\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-kube-api-access-c59fb\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.161713 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-envoy\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.161713 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-data\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.161713 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-podinfo\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.162014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istiod-ca-cert\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.162014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-socket\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.162014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.161960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-certs\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.162325 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.162301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-data\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.162579 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.162538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istiod-ca-cert\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.162752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.162712 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-credential-socket\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.164783 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.164761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-envoy\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.165473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.165451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-podinfo\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.169968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.169948 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-token\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.170806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.170787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59fb\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-kube-api-access-c59fb\") pod \"e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.305046 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.304776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:44.618424 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.617804 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht"] Sep 07 10:24:44.620748 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:24:44.620717 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81003c8_f0db_4886_9c81_ab16bf4a4b02.slice/crio-fd45dd17130b742a539cba70cb5e40aa74e1df237920075f699abb1017e96267 WatchSource:0}: Error finding container fd45dd17130b742a539cba70cb5e40aa74e1df237920075f699abb1017e96267: Status 404 returned error can't find the container with id fd45dd17130b742a539cba70cb5e40aa74e1df237920075f699abb1017e96267 Sep 07 10:24:44.623458 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.623423 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:44.623559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.623506 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:44.623559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.623550 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:44.665231 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:44.665192 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" event={"ID":"a81003c8-f0db-4886-9c81-ab16bf4a4b02","Type":"ContainerStarted","Data":"fd45dd17130b742a539cba70cb5e40aa74e1df237920075f699abb1017e96267"} Sep 07 10:24:45.044627 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.044587 2565 patch_prober.go:28] interesting pod/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:24:45.044826 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.044658 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 07 10:24:45.671898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.671861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" event={"ID":"a81003c8-f0db-4886-9c81-ab16bf4a4b02","Type":"ContainerStarted","Data":"eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb"} Sep 07 10:24:45.673585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.673561 2565 generic.go:356] "Generic (PLEG): container finished" podID="4576c52d-4883-44c1-b5a4-478251329113" containerID="ab448cacd7d0f1d70dacd9cc313db0d3cc8d2f81ae8d926dac49d68301f9fd10" exitCode=0 Sep 07 10:24:45.673722 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.673635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" event={"ID":"4576c52d-4883-44c1-b5a4-478251329113","Type":"ContainerDied","Data":"ab448cacd7d0f1d70dacd9cc313db0d3cc8d2f81ae8d926dac49d68301f9fd10"} Sep 07 10:24:45.674044 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.674030 2565 scope.go:117] "RemoveContainer" containerID="ab448cacd7d0f1d70dacd9cc313db0d3cc8d2f81ae8d926dac49d68301f9fd10" Sep 07 10:24:45.675073 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.675047 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:45.675149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.675111 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:45.675149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.675144 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:45.693695 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:45.693653 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" podStartSLOduration=2.69363191 podStartE2EDuration="2.69363191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:45.692598918 +0000 UTC m=+807.678879253" watchObservedRunningTime="2026-09-07 10:24:45.69363191 +0000 UTC m=+807.679912243" Sep 07 10:24:46.044740 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.044697 2565 patch_prober.go:28] interesting pod/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:24:46.045980 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.044765 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 07 10:24:46.305122 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.304976 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:46.312249 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.312221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:46.682756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.682657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" event={"ID":"4576c52d-4883-44c1-b5a4-478251329113","Type":"ContainerStarted","Data":"43a1e705502ae7f6777a1e503115b628c8e0cb39469317bd5c0054cd8feb684b"} Sep 07 10:24:46.685138 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.685111 2565 generic.go:356] "Generic (PLEG): container finished" podID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerID="b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131" exitCode=0 Sep 07 10:24:46.685285 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.685194 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" event={"ID":"426da8e4-5ecc-4da2-89ef-b695d8cc17ce","Type":"ContainerDied","Data":"b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131"} Sep 07 10:24:46.685455 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.685436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:46.685667 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.685640 2565 scope.go:117] "RemoveContainer" containerID="b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131" Sep 07 10:24:46.686537 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.686380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:24:46.686850 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.686827 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:46.686904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.686886 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:46.686942 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:46.686918 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:24:47.054610 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.054567 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:47.055942 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.055907 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:47.060529 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.060504 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:24:47.498859 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.498777 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:47.499038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.498951 2565 patch_prober.go:28] interesting pod/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 07 10:24:47.499038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.498996 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 07 10:24:47.693934 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.693900 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" event={"ID":"426da8e4-5ecc-4da2-89ef-b695d8cc17ce","Type":"ContainerStarted","Data":"2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b"} Sep 07 10:24:47.741739 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.741690 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:47.742098 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.742051 2565 patch_prober.go:28] interesting pod/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 07 10:24:47.742210 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:47.742134 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 07 10:24:48.498005 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:48.497963 2565 patch_prober.go:28] interesting pod/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 07 10:24:48.498219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:48.498029 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 07 10:24:48.742242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:48.742204 2565 patch_prober.go:28] interesting pod/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 07 10:24:48.742736 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:48.742285 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 07 10:24:49.498959 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:49.498807 2565 patch_prober.go:28] interesting pod/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 07 10:24:49.498959 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:49.498862 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 07 10:24:49.759250 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:49.757561 2565 patch_prober.go:28] interesting pod/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 07 10:24:49.759250 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:49.757626 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 07 10:24:50.498637 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:50.498601 2565 patch_prober.go:28] interesting pod/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 07 10:24:50.498826 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:50.498656 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 07 10:24:50.742495 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:50.742450 2565 patch_prober.go:28] interesting pod/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 07 10:24:50.742693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:50.742520 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 07 10:24:51.319871 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.319841 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt"] Sep 07 10:24:51.323220 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.323204 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" Sep 07 10:24:51.325968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.325948 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:24:51.325968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.325957 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:24:51.332103 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.331692 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt"] Sep 07 10:24:51.455184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.453773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b804caf0-5a94-4c62-9cba-029776562c55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt\" (UID: \"b804caf0-5a94-4c62-9cba-029776562c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" Sep 07 10:24:51.499007 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.498970 2565 patch_prober.go:28] interesting pod/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 07 10:24:51.499281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.499043 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 07 10:24:51.555238 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.555198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b804caf0-5a94-4c62-9cba-029776562c55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt\" (UID: \"b804caf0-5a94-4c62-9cba-029776562c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" Sep 07 10:24:51.556062 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.556029 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b804caf0-5a94-4c62-9cba-029776562c55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt\" (UID: \"b804caf0-5a94-4c62-9cba-029776562c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" Sep 07 10:24:51.636661 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.636565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" Sep 07 10:24:51.742184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.741798 2565 patch_prober.go:28] interesting pod/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 07 10:24:51.742184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:51.741856 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 07 10:24:52.025074 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:52.025049 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt"] Sep 07 10:24:52.027404 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:24:52.027365 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb804caf0_5a94_4c62_9cba_029776562c55.slice/crio-5583c036d9738576407333eecbc5f8ab3eb8d2ab0886134a77c6926e34692e45 WatchSource:0}: Error finding container 5583c036d9738576407333eecbc5f8ab3eb8d2ab0886134a77c6926e34692e45: Status 404 returned error can't find the container with id 5583c036d9738576407333eecbc5f8ab3eb8d2ab0886134a77c6926e34692e45 Sep 07 10:24:52.502493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:52.502459 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:52.502949 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:52.502755 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:52.503449 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:52.503435 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:24:52.737736 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:52.737688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" event={"ID":"b804caf0-5a94-4c62-9cba-029776562c55","Type":"ContainerStarted","Data":"5583c036d9738576407333eecbc5f8ab3eb8d2ab0886134a77c6926e34692e45"} Sep 07 10:24:52.742174 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:52.742127 2565 patch_prober.go:28] interesting pod/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 07 10:24:52.742297 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:52.742202 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 07 10:24:53.746359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:53.746332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:53.746814 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:53.746718 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:53.747569 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:53.747548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:24:55.757585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:55.757550 2565 generic.go:356] "Generic (PLEG): container finished" podID="b804caf0-5a94-4c62-9cba-029776562c55" containerID="975c9127d60f069992e76165b1fded2b2bfc38bfd7b0b9d5fcbcdc898ea52465" exitCode=0 Sep 07 10:24:55.758207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:55.757632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" event={"ID":"b804caf0-5a94-4c62-9cba-029776562c55","Type":"ContainerDied","Data":"975c9127d60f069992e76165b1fded2b2bfc38bfd7b0b9d5fcbcdc898ea52465"} Sep 07 10:24:56.893849 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:56.893825 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" Sep 07 10:24:57.022802 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:57.022718 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b804caf0-5a94-4c62-9cba-029776562c55-maas-api-service-ca\") pod \"b804caf0-5a94-4c62-9cba-029776562c55\" (UID: \"b804caf0-5a94-4c62-9cba-029776562c55\") " Sep 07 10:24:57.023073 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:57.023052 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b804caf0-5a94-4c62-9cba-029776562c55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b804caf0-5a94-4c62-9cba-029776562c55" (UID: "b804caf0-5a94-4c62-9cba-029776562c55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:24:57.124445 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:57.124398 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b804caf0-5a94-4c62-9cba-029776562c55-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:24:57.769535 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:57.769506 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" Sep 07 10:24:57.769727 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:57.769540 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt" event={"ID":"b804caf0-5a94-4c62-9cba-029776562c55","Type":"ContainerDied","Data":"5583c036d9738576407333eecbc5f8ab3eb8d2ab0886134a77c6926e34692e45"} Sep 07 10:24:57.769727 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:24:57.769589 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5583c036d9738576407333eecbc5f8ab3eb8d2ab0886134a77c6926e34692e45" Sep 07 10:25:08.208392 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:08.208354 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt"] Sep 07 10:25:08.210536 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:08.210506 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ec6b6b5-vltkt"] Sep 07 10:25:08.333345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:08.333313 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l"] Sep 07 10:25:08.333831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:08.333783 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" containerID="cri-o://2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b" gracePeriod=30 Sep 07 10:25:08.544418 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:08.544385 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b804caf0-5a94-4c62-9cba-029776562c55" path="/var/lib/kubelet/pods/b804caf0-5a94-4c62-9cba-029776562c55/volumes" Sep 07 10:25:08.749819 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:08.749782 2565 patch_prober.go:28] interesting pod/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 07 10:25:08.750021 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:08.749848 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:25:09.227987 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.227955 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5"] Sep 07 10:25:09.229444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.228419 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b804caf0-5a94-4c62-9cba-029776562c55" containerName="revoke-keys" Sep 07 10:25:09.229444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.228432 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b804caf0-5a94-4c62-9cba-029776562c55" containerName="revoke-keys" Sep 07 10:25:09.229444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.228533 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b804caf0-5a94-4c62-9cba-029776562c55" containerName="revoke-keys" Sep 07 10:25:09.230401 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.230382 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.233139 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.233121 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-de6561a9-gw-openshift-default-dockercfg-l8vxl\"" Sep 07 10:25:09.246139 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.246112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5"] Sep 07 10:25:09.346616 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-data\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.346804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-istio-token\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.346804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346697 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-certs\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.346804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-credential-socket\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.346956 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-envoy\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.346956 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346837 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-socket\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.346956 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzf4\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-kube-api-access-8dzf4\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.346956 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797d4337-962d-4f40-acc0-a13d79c65d84-istio-podinfo\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.347131 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.346966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797d4337-962d-4f40-acc0-a13d79c65d84-istiod-ca-cert\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.447911 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.447825 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-certs\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.447911 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.447868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-credential-socket\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448153 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.447922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-envoy\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448153 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.447942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-socket\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448153 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.447983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzf4\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-kube-api-access-8dzf4\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797d4337-962d-4f40-acc0-a13d79c65d84-istio-podinfo\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797d4337-962d-4f40-acc0-a13d79c65d84-istiod-ca-cert\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-certs\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-data\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-credential-socket\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-istio-token\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448648 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448372 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-socket\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448648 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-data\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.448922 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.448870 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797d4337-962d-4f40-acc0-a13d79c65d84-istiod-ca-cert\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.450452 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.450430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-envoy\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.450732 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.450714 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797d4337-962d-4f40-acc0-a13d79c65d84-istio-podinfo\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.462072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.462043 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-istio-token\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.462305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.462284 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzf4\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-kube-api-access-8dzf4\") pod \"e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.544760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.544680 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:09.822631 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.821827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5"] Sep 07 10:25:09.829573 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.829534 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:09.829707 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.829608 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:09.829707 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.829650 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:09.834913 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.834893 2565 generic.go:356] "Generic (PLEG): container finished" podID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerID="eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb" exitCode=0 Sep 07 10:25:09.835022 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.834942 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" event={"ID":"a81003c8-f0db-4886-9c81-ab16bf4a4b02","Type":"ContainerDied","Data":"eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb"} Sep 07 10:25:09.835436 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.835417 2565 scope.go:117] "RemoveContainer" containerID="eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb" Sep 07 10:25:09.836305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.836284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" event={"ID":"797d4337-962d-4f40-acc0-a13d79c65d84","Type":"ContainerStarted","Data":"64395574b42cec2b87265d225fc4187a2e9bcbea3a4897f20e2c23a3653b94f0"} Sep 07 10:25:09.839263 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.839235 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:09.839335 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.839295 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:09.839375 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:09.839347 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:10.843364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:10.843333 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" event={"ID":"a81003c8-f0db-4886-9c81-ab16bf4a4b02","Type":"ContainerStarted","Data":"fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4"} Sep 07 10:25:10.844971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:10.844948 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" event={"ID":"797d4337-962d-4f40-acc0-a13d79c65d84","Type":"ContainerStarted","Data":"579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce"} Sep 07 10:25:10.885676 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:10.885617 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" podStartSLOduration=1.885598702 podStartE2EDuration="1.885598702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:10.882893072 +0000 UTC m=+832.869173419" watchObservedRunningTime="2026-09-07 10:25:10.885598702 +0000 UTC m=+832.871879035" Sep 07 10:25:11.305132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:11.305098 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:25:11.305431 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:11.305409 2565 patch_prober.go:28] interesting pod/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 07 10:25:11.305487 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:11.305462 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 07 10:25:11.546552 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:11.546189 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:11.558375 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:11.558283 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:11.855807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:11.855712 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:11.859147 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:11.859122 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:12.305417 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:12.305380 2565 patch_prober.go:28] interesting pod/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 07 10:25:12.305594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:12.305455 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 07 10:25:13.305484 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.305448 2565 patch_prober.go:28] interesting pod/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 07 10:25:13.305870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.305502 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 07 10:25:13.627176 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.627140 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:25:13.696076 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696040 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-socket\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696290 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istiod-ca-cert\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696290 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696276 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-certs\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696301 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-podinfo\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696328 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-credential-socket\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696331 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:13.696415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696363 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-envoy\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696443 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-data\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696471 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z54gt\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-kube-api-access-z54gt\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696499 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-token\") pod \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\" (UID: \"426da8e4-5ecc-4da2-89ef-b695d8cc17ce\") " Sep 07 10:25:13.696613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696554 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:25:13.696613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696557 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:13.696862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696744 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:13.696917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696887 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.696917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696905 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.697136 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696923 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.697136 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696940 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.697136 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.696964 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:13.698715 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.698692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:13.698715 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.698699 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-kube-api-access-z54gt" (OuterVolumeSpecName: "kube-api-access-z54gt") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "kube-api-access-z54gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:25:13.698862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.698742 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:25:13.698862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.698815 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "426da8e4-5ecc-4da2-89ef-b695d8cc17ce" (UID: "426da8e4-5ecc-4da2-89ef-b695d8cc17ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:25:13.797737 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.797706 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.797737 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.797732 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.797737 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.797741 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.797971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.797750 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54gt\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-kube-api-access-z54gt\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.797971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.797759 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426da8e4-5ecc-4da2-89ef-b695d8cc17ce-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:13.884808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.884723 2565 generic.go:356] "Generic (PLEG): container finished" podID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerID="2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b" exitCode=0 Sep 07 10:25:13.884808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.884775 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" Sep 07 10:25:13.884989 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.884819 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" event={"ID":"426da8e4-5ecc-4da2-89ef-b695d8cc17ce","Type":"ContainerDied","Data":"2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b"} Sep 07 10:25:13.884989 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.884896 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l" event={"ID":"426da8e4-5ecc-4da2-89ef-b695d8cc17ce","Type":"ContainerDied","Data":"b0686cc31fd5372906c9d4c9418639116dc1d11e27fd793ac71753f8ff94ed1e"} Sep 07 10:25:13.884989 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.884920 2565 scope.go:117] "RemoveContainer" containerID="2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b" Sep 07 10:25:13.894877 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.894860 2565 scope.go:117] "RemoveContainer" containerID="b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131" Sep 07 10:25:13.903200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.903183 2565 scope.go:117] "RemoveContainer" containerID="2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b" Sep 07 10:25:13.903544 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:25:13.903518 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b\": container with ID starting with 2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b not found: ID does not exist" containerID="2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b" Sep 07 10:25:13.903643 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.903555 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b"} err="failed to get container status \"2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b\": rpc error: code = NotFound desc = could not find container \"2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b\": container with ID starting with 2e09dc68ead14bb30959a2d50ede4c4eda9121ffa5890e2a61e20b75f8c9437b not found: ID does not exist" Sep 07 10:25:13.903643 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.903577 2565 scope.go:117] "RemoveContainer" containerID="b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131" Sep 07 10:25:13.904347 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:25:13.904279 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131\": container with ID starting with b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131 not found: ID does not exist" containerID="b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131" Sep 07 10:25:13.904347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.904310 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131"} err="failed to get container status \"b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131\": rpc error: code = NotFound desc = could not find container \"b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131\": container with ID starting with b16f0d33404b6163f01b8459531ccfbd88b5cf4a4d53de210e1700657a298131 not found: ID does not exist" Sep 07 10:25:13.906877 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.906859 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l"] Sep 07 10:25:13.910053 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:13.910035 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8ec6b6b5-openshift-default-657f7d5d6b-gts9l"] Sep 07 10:25:14.305909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:14.305869 2565 patch_prober.go:28] interesting pod/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 07 10:25:14.306341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:14.305935 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 07 10:25:14.549551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:14.549504 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" path="/var/lib/kubelet/pods/426da8e4-5ecc-4da2-89ef-b695d8cc17ce/volumes" Sep 07 10:25:15.306123 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:15.306093 2565 patch_prober.go:28] interesting pod/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 07 10:25:15.306508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:15.306183 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 07 10:25:16.305993 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:16.305958 2565 patch_prober.go:28] interesting pod/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 07 10:25:16.306200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:16.306014 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 07 10:25:16.686137 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:16.686050 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:25:17.309730 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:17.309704 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:25:17.310809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:17.310783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:25:33.006814 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:33.006774 2565 generic.go:356] "Generic (PLEG): container finished" podID="797d4337-962d-4f40-acc0-a13d79c65d84" containerID="579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce" exitCode=0 Sep 07 10:25:33.007259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:33.006847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" event={"ID":"797d4337-962d-4f40-acc0-a13d79c65d84","Type":"ContainerDied","Data":"579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce"} Sep 07 10:25:33.007259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:33.007246 2565 scope.go:117] "RemoveContainer" containerID="579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce" Sep 07 10:25:33.008314 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:33.008286 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:33.008375 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:33.008352 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:33.008416 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:33.008385 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:34.014692 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:34.014659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" event={"ID":"797d4337-962d-4f40-acc0-a13d79c65d84","Type":"ContainerStarted","Data":"de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee"} Sep 07 10:25:34.545273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:34.545236 2565 patch_prober.go:28] interesting pod/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:25:34.545491 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:34.545301 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 07 10:25:34.548612 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:34.548582 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:35.545827 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:35.545784 2565 patch_prober.go:28] interesting pod/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:25:35.546364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:35.545852 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 07 10:25:36.546786 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:36.546744 2565 patch_prober.go:28] interesting pod/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:25:36.547337 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:36.546810 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 07 10:25:37.220888 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.220852 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527"] Sep 07 10:25:37.221289 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.221277 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" Sep 07 10:25:37.221353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.221291 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" Sep 07 10:25:37.221396 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.221381 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" Sep 07 10:25:37.221396 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.221390 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" Sep 07 10:25:37.224564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.224549 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" Sep 07 10:25:37.227001 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.226979 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:25:37.227001 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.226997 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:25:37.231489 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.231464 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527"] Sep 07 10:25:37.246184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.242209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fb8771-d1d2-4c80-a90a-c913ceceda22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de6561a9-zm527\" (UID: \"31fb8771-d1d2-4c80-a90a-c913ceceda22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" Sep 07 10:25:37.345185 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.343831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fb8771-d1d2-4c80-a90a-c913ceceda22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de6561a9-zm527\" (UID: \"31fb8771-d1d2-4c80-a90a-c913ceceda22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" Sep 07 10:25:37.345185 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.344769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fb8771-d1d2-4c80-a90a-c913ceceda22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-de6561a9-zm527\" (UID: \"31fb8771-d1d2-4c80-a90a-c913ceceda22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" Sep 07 10:25:37.537208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.537068 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" Sep 07 10:25:37.551010 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.550979 2565 patch_prober.go:28] interesting pod/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:25:37.552451 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.552418 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 07 10:25:37.831363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:37.831340 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527"] Sep 07 10:25:37.833753 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:25:37.833707 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fb8771_d1d2_4c80_a90a_c913ceceda22.slice/crio-c08d588bda1953358845e6035e407eed44e0e12f4667afcaeda6b806cbbfeff1 WatchSource:0}: Error finding container c08d588bda1953358845e6035e407eed44e0e12f4667afcaeda6b806cbbfeff1: Status 404 returned error can't find the container with id c08d588bda1953358845e6035e407eed44e0e12f4667afcaeda6b806cbbfeff1 Sep 07 10:25:38.039783 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:38.039745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" event={"ID":"31fb8771-d1d2-4c80-a90a-c913ceceda22","Type":"ContainerStarted","Data":"c08d588bda1953358845e6035e407eed44e0e12f4667afcaeda6b806cbbfeff1"} Sep 07 10:25:38.546801 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:38.546717 2565 patch_prober.go:28] interesting pod/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:25:38.547032 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:38.547007 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 07 10:25:39.051335 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:39.051291 2565 generic.go:356] "Generic (PLEG): container finished" podID="31fb8771-d1d2-4c80-a90a-c913ceceda22" containerID="ed050a72522192454f9f2226d8fe969c5afe97d623f6e0db3ee3cb367a49bd62" exitCode=0 Sep 07 10:25:39.051820 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:39.051521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" event={"ID":"31fb8771-d1d2-4c80-a90a-c913ceceda22","Type":"ContainerDied","Data":"ed050a72522192454f9f2226d8fe969c5afe97d623f6e0db3ee3cb367a49bd62"} Sep 07 10:25:39.545316 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:39.545270 2565 patch_prober.go:28] interesting pod/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:25:39.545480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:39.545342 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 07 10:25:40.200429 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:40.200405 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" Sep 07 10:25:40.282352 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:40.282319 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fb8771-d1d2-4c80-a90a-c913ceceda22-maas-api-service-ca\") pod \"31fb8771-d1d2-4c80-a90a-c913ceceda22\" (UID: \"31fb8771-d1d2-4c80-a90a-c913ceceda22\") " Sep 07 10:25:40.282687 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:40.282665 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fb8771-d1d2-4c80-a90a-c913ceceda22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31fb8771-d1d2-4c80-a90a-c913ceceda22" (UID: "31fb8771-d1d2-4c80-a90a-c913ceceda22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:25:40.383316 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:40.383219 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31fb8771-d1d2-4c80-a90a-c913ceceda22-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:40.549631 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:40.549603 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:40.550046 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:40.550023 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:40.550860 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:40.550837 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:41.064205 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:41.064173 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" Sep 07 10:25:41.064205 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:41.064198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527" event={"ID":"31fb8771-d1d2-4c80-a90a-c913ceceda22","Type":"ContainerDied","Data":"c08d588bda1953358845e6035e407eed44e0e12f4667afcaeda6b806cbbfeff1"} Sep 07 10:25:41.064477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:41.064249 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c08d588bda1953358845e6035e407eed44e0e12f4667afcaeda6b806cbbfeff1" Sep 07 10:25:43.283713 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.283679 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58"] Sep 07 10:25:43.284220 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.284203 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31fb8771-d1d2-4c80-a90a-c913ceceda22" containerName="revoke-keys" Sep 07 10:25:43.284220 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.284221 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31fb8771-d1d2-4c80-a90a-c913ceceda22" containerName="revoke-keys" Sep 07 10:25:43.284333 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.284231 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" Sep 07 10:25:43.284333 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.284237 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="426da8e4-5ecc-4da2-89ef-b695d8cc17ce" containerName="istio-proxy" Sep 07 10:25:43.284333 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.284309 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="31fb8771-d1d2-4c80-a90a-c913ceceda22" containerName="revoke-keys" Sep 07 10:25:43.287429 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.287412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.292150 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.292130 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ff5470\"/\"autores-model-ff5470-kserve-self-signed-certs\"" Sep 07 10:25:43.293193 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.293152 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ff5470\"/\"kube-root-ca.crt\"" Sep 07 10:25:43.293304 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.293233 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ff5470\"/\"openshift-service-ca.crt\"" Sep 07 10:25:43.293304 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.293280 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ff5470\"/\"default-dockercfg-2njpz\"" Sep 07 10:25:43.298210 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.298187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58"] Sep 07 10:25:43.311834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.311808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-dshm\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.312006 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.311877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e599d6a-8772-4d73-bfba-7c579832b0b1-tls-certs\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.312006 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.311941 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-model-cache\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.312006 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.311980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-tmp-dir\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.312214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.312034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4hk\" (UniqueName: \"kubernetes.io/projected/3e599d6a-8772-4d73-bfba-7c579832b0b1-kube-api-access-5w4hk\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.312214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.312112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-home\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.413598 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.413562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-home\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.413804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.413622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-dshm\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.413804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.413676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e599d6a-8772-4d73-bfba-7c579832b0b1-tls-certs\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.413804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.413699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-model-cache\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.413804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.413714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-tmp-dir\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.413804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.413742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4hk\" (UniqueName: \"kubernetes.io/projected/3e599d6a-8772-4d73-bfba-7c579832b0b1-kube-api-access-5w4hk\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.414070 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.414015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-home\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.414113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.414092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-model-cache\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.414149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.414116 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-tmp-dir\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.416044 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.416022 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-dshm\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.416365 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.416348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e599d6a-8772-4d73-bfba-7c579832b0b1-tls-certs\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.421971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.421953 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4hk\" (UniqueName: \"kubernetes.io/projected/3e599d6a-8772-4d73-bfba-7c579832b0b1-kube-api-access-5w4hk\") pod \"autores-model-ff5470-kserve-5f84fc5c4c-4kd58\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.598587 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.598498 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:43.886688 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:43.886657 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58"] Sep 07 10:25:43.888203 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:25:43.888181 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e599d6a_8772_4d73_bfba_7c579832b0b1.slice/crio-e557d2ac663ecfd73d00a5c0809fbd9de3f1bec8c11ba795c7da16e12f16068c WatchSource:0}: Error finding container e557d2ac663ecfd73d00a5c0809fbd9de3f1bec8c11ba795c7da16e12f16068c: Status 404 returned error can't find the container with id e557d2ac663ecfd73d00a5c0809fbd9de3f1bec8c11ba795c7da16e12f16068c Sep 07 10:25:44.082320 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.082284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" event={"ID":"3e599d6a-8772-4d73-bfba-7c579832b0b1","Type":"ContainerStarted","Data":"c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b"} Sep 07 10:25:44.082483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.082332 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" event={"ID":"3e599d6a-8772-4d73-bfba-7c579832b0b1","Type":"ContainerStarted","Data":"e557d2ac663ecfd73d00a5c0809fbd9de3f1bec8c11ba795c7da16e12f16068c"} Sep 07 10:25:44.102500 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.102448 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" podStartSLOduration=1.102433548 podStartE2EDuration="1.102433548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:44.10094595 +0000 UTC m=+866.087226297" watchObservedRunningTime="2026-09-07 10:25:44.102433548 +0000 UTC m=+866.088713882" Sep 07 10:25:44.174911 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.174823 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527"] Sep 07 10:25:44.177002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.176980 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-de6561a9-zm527"] Sep 07 10:25:44.304881 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.304852 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5"] Sep 07 10:25:44.305354 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.305136 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" containerID="cri-o://de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee" gracePeriod=30 Sep 07 10:25:44.544425 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.544396 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31fb8771-d1d2-4c80-a90a-c913ceceda22" path="/var/lib/kubelet/pods/31fb8771-d1d2-4c80-a90a-c913ceceda22/volumes" Sep 07 10:25:44.990414 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.990387 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n"] Sep 07 10:25:44.994038 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.994018 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:44.996614 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:44.996597 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-31186eae-openshift-default-dockercfg-v8ps4\"" Sep 07 10:25:45.006895 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.006870 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n"] Sep 07 10:25:45.034311 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qjs\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-kube-api-access-95qjs\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.034311 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-socket\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.034571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-envoy\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.034571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034498 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-podinfo\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.034571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-credential-socket\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.034571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034565 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82b3aeb-bd27-42d8-b602-296f533d0702-istiod-ca-cert\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.034746 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-token\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.034746 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-data\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.034746 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.034712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-certs\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135343 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-certs\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135539 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qjs\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-kube-api-access-95qjs\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135539 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-socket\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135668 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-envoy\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-podinfo\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-credential-socket\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135839 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82b3aeb-bd27-42d8-b602-296f533d0702-istiod-ca-cert\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135839 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135749 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-token\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135839 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-certs\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.135839 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-data\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.136030 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.135876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-socket\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.136094 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.136012 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-credential-socket\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.136250 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.136221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-data\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.136562 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.136539 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82b3aeb-bd27-42d8-b602-296f533d0702-istiod-ca-cert\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.143195 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.138999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-envoy\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.144487 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.144455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-podinfo\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.151179 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.150429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-token\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.152387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.151985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qjs\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-kube-api-access-95qjs\") pod \"e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.308415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.308314 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:45.590888 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.590777 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n"] Sep 07 10:25:45.597621 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:25:45.597586 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82b3aeb_bd27_42d8_b602_296f533d0702.slice/crio-537d04f8e8f39e63d74125853c21a3b87043efca9a1a6c1655b8d5c9e91dc491 WatchSource:0}: Error finding container 537d04f8e8f39e63d74125853c21a3b87043efca9a1a6c1655b8d5c9e91dc491: Status 404 returned error can't find the container with id 537d04f8e8f39e63d74125853c21a3b87043efca9a1a6c1655b8d5c9e91dc491 Sep 07 10:25:45.600113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.600075 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:45.600258 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.600155 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:45.600258 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:45.600218 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:25:46.094066 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:46.094029 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" event={"ID":"f82b3aeb-bd27-42d8-b602-296f533d0702","Type":"ContainerStarted","Data":"65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845"} Sep 07 10:25:46.094268 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:46.094086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" event={"ID":"f82b3aeb-bd27-42d8-b602-296f533d0702","Type":"ContainerStarted","Data":"537d04f8e8f39e63d74125853c21a3b87043efca9a1a6c1655b8d5c9e91dc491"} Sep 07 10:25:46.117461 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:46.117404 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" podStartSLOduration=2.117387591 podStartE2EDuration="2.117387591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:46.115643178 +0000 UTC m=+868.101923518" watchObservedRunningTime="2026-09-07 10:25:46.117387591 +0000 UTC m=+868.103667926" Sep 07 10:25:46.308795 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:46.308701 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:46.317813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:46.317775 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:47.099146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:47.099117 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:47.100187 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:47.100140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:25:49.598031 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.597967 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:49.686169 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686133 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-data\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686242 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-istio-token\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686283 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-credential-socket\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686308 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-socket\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686377 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797d4337-962d-4f40-acc0-a13d79c65d84-istio-podinfo\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686405 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-data" (OuterVolumeSpecName: "istio-data") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:49.686477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686413 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dzf4\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-kube-api-access-8dzf4\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686458 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797d4337-962d-4f40-acc0-a13d79c65d84-istiod-ca-cert\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686483 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-certs\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686501 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:49.686689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686591 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:49.686689 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686596 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-envoy\") pod \"797d4337-962d-4f40-acc0-a13d79c65d84\" (UID: \"797d4337-962d-4f40-acc0-a13d79c65d84\") " Sep 07 10:25:49.686892 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686768 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797d4337-962d-4f40-acc0-a13d79c65d84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:25:49.686892 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686771 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:49.686996 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686980 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:49.687050 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.686999 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:49.687050 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.687008 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:49.687050 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.687016 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797d4337-962d-4f40-acc0-a13d79c65d84-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:49.687050 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.687029 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:49.689150 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.689120 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/797d4337-962d-4f40-acc0-a13d79c65d84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:25:49.689150 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.689138 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-kube-api-access-8dzf4" (OuterVolumeSpecName: "kube-api-access-8dzf4") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "kube-api-access-8dzf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:25:49.689388 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.689367 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:25:49.689443 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.689389 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-istio-token" (OuterVolumeSpecName: "istio-token") pod "797d4337-962d-4f40-acc0-a13d79c65d84" (UID: "797d4337-962d-4f40-acc0-a13d79c65d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:25:49.788363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.788334 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797d4337-962d-4f40-acc0-a13d79c65d84-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:49.788551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.788426 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzf4\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-kube-api-access-8dzf4\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:49.788551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.788445 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797d4337-962d-4f40-acc0-a13d79c65d84-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:49.788551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:49.788457 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797d4337-962d-4f40-acc0-a13d79c65d84-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:25:50.116534 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.116499 2565 generic.go:356] "Generic (PLEG): container finished" podID="797d4337-962d-4f40-acc0-a13d79c65d84" containerID="de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee" exitCode=0 Sep 07 10:25:50.116722 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.116558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" event={"ID":"797d4337-962d-4f40-acc0-a13d79c65d84","Type":"ContainerDied","Data":"de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee"} Sep 07 10:25:50.116722 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.116562 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" Sep 07 10:25:50.116722 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.116638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5" event={"ID":"797d4337-962d-4f40-acc0-a13d79c65d84","Type":"ContainerDied","Data":"64395574b42cec2b87265d225fc4187a2e9bcbea3a4897f20e2c23a3653b94f0"} Sep 07 10:25:50.116722 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.116660 2565 scope.go:117] "RemoveContainer" containerID="de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee" Sep 07 10:25:50.126430 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.126408 2565 scope.go:117] "RemoveContainer" containerID="579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce" Sep 07 10:25:50.134949 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.134924 2565 scope.go:117] "RemoveContainer" containerID="de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee" Sep 07 10:25:50.135377 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:25:50.135349 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee\": container with ID starting with de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee not found: ID does not exist" containerID="de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee" Sep 07 10:25:50.135491 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.135386 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee"} err="failed to get container status \"de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee\": rpc error: code = NotFound desc = could not find container \"de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee\": container with ID starting with de8185ad5f362b6156fa2caa2ffdde9f74e1c56b377a305d95134b69900cd2ee not found: ID does not exist" Sep 07 10:25:50.135491 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.135409 2565 scope.go:117] "RemoveContainer" containerID="579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce" Sep 07 10:25:50.135723 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:25:50.135708 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce\": container with ID starting with 579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce not found: ID does not exist" containerID="579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce" Sep 07 10:25:50.135763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.135728 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce"} err="failed to get container status \"579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce\": rpc error: code = NotFound desc = could not find container \"579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce\": container with ID starting with 579719699c2850f0b3f2f54e835bec160bbdbad749e8b6b89f63ff59835dddce not found: ID does not exist" Sep 07 10:25:50.137860 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.137838 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5"] Sep 07 10:25:50.140714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.140693 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-de6561a9-gw-openshift-default-765dfcdfd8-7tjp5"] Sep 07 10:25:50.544563 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:50.544533 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" path="/var/lib/kubelet/pods/797d4337-962d-4f40-acc0-a13d79c65d84/volumes" Sep 07 10:25:53.598998 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:53.598963 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:53.599405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:53.599016 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:53.611557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:53.611533 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:25:54.152342 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:25:54.152316 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:26:12.249303 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.249271 2565 generic.go:356] "Generic (PLEG): container finished" podID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerID="65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845" exitCode=0 Sep 07 10:26:12.249753 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.249321 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" event={"ID":"f82b3aeb-bd27-42d8-b602-296f533d0702","Type":"ContainerDied","Data":"65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845"} Sep 07 10:26:12.249753 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.249737 2565 scope.go:117] "RemoveContainer" containerID="65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845" Sep 07 10:26:12.250810 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.250776 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:12.250920 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.250845 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:12.250920 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.250880 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:12.672013 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.671929 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58"] Sep 07 10:26:12.793432 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.793393 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x"] Sep 07 10:26:12.793921 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.793901 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" Sep 07 10:26:12.794026 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.793923 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" Sep 07 10:26:12.794026 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.793945 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" Sep 07 10:26:12.794026 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.793952 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" Sep 07 10:26:12.800604 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.794703 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" Sep 07 10:26:12.801054 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.800822 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" Sep 07 10:26:12.804148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.804120 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:26:12.804278 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.804120 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:26:12.804584 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.804562 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x"] Sep 07 10:26:12.837151 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.837111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc59765-bbdd-48aa-9bc9-5c541a4987d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ff5470-8mz9x\" (UID: \"fcc59765-bbdd-48aa-9bc9-5c541a4987d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" Sep 07 10:26:12.938793 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.938715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc59765-bbdd-48aa-9bc9-5c541a4987d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ff5470-8mz9x\" (UID: \"fcc59765-bbdd-48aa-9bc9-5c541a4987d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" Sep 07 10:26:12.939370 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:12.939352 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc59765-bbdd-48aa-9bc9-5c541a4987d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ff5470-8mz9x\" (UID: \"fcc59765-bbdd-48aa-9bc9-5c541a4987d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" Sep 07 10:26:13.114560 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.114257 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" Sep 07 10:26:13.178739 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.178703 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw"] Sep 07 10:26:13.179352 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.179333 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="797d4337-962d-4f40-acc0-a13d79c65d84" containerName="istio-proxy" Sep 07 10:26:13.182850 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.182833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" Sep 07 10:26:13.189244 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.187635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw"] Sep 07 10:26:13.256185 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.243793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da52000-a13f-44f3-aa14-7f10d4b44089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-31186eae-fpxlw\" (UID: \"7da52000-a13f-44f3-aa14-7f10d4b44089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" Sep 07 10:26:13.266188 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.266133 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" event={"ID":"f82b3aeb-bd27-42d8-b602-296f533d0702","Type":"ContainerStarted","Data":"c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3"} Sep 07 10:26:13.309022 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.308988 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:26:13.310109 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.310084 2565 patch_prober.go:28] interesting pod/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 07 10:26:13.310251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.310140 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 07 10:26:13.352186 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.351784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da52000-a13f-44f3-aa14-7f10d4b44089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-31186eae-fpxlw\" (UID: \"7da52000-a13f-44f3-aa14-7f10d4b44089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" Sep 07 10:26:13.355192 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.352930 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da52000-a13f-44f3-aa14-7f10d4b44089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-31186eae-fpxlw\" (UID: \"7da52000-a13f-44f3-aa14-7f10d4b44089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" Sep 07 10:26:13.411606 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.411575 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x"] Sep 07 10:26:13.414235 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:26:13.414206 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc59765_bbdd_48aa_9bc9_5c541a4987d4.slice/crio-bf53d7ad21b31affc2204d9681a67491132bb45b6924578c2b3c82f5a2521160 WatchSource:0}: Error finding container bf53d7ad21b31affc2204d9681a67491132bb45b6924578c2b3c82f5a2521160: Status 404 returned error can't find the container with id bf53d7ad21b31affc2204d9681a67491132bb45b6924578c2b3c82f5a2521160 Sep 07 10:26:13.498889 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.498859 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" Sep 07 10:26:13.793145 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:13.793121 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw"] Sep 07 10:26:13.794972 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:26:13.794942 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da52000_a13f_44f3_aa14_7f10d4b44089.slice/crio-610d8304697d24a0e90aa0fa88fc823cfc1991778ec8750d9a44de57d2d72253 WatchSource:0}: Error finding container 610d8304697d24a0e90aa0fa88fc823cfc1991778ec8750d9a44de57d2d72253: Status 404 returned error can't find the container with id 610d8304697d24a0e90aa0fa88fc823cfc1991778ec8750d9a44de57d2d72253 Sep 07 10:26:14.277090 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:14.277057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" event={"ID":"fcc59765-bbdd-48aa-9bc9-5c541a4987d4","Type":"ContainerStarted","Data":"a8905c5b2bef80dc2c6230ffc9560360a7126a352f7d8c0898504fe1eea68c46"} Sep 07 10:26:14.277725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:14.277656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" event={"ID":"fcc59765-bbdd-48aa-9bc9-5c541a4987d4","Type":"ContainerStarted","Data":"bf53d7ad21b31affc2204d9681a67491132bb45b6924578c2b3c82f5a2521160"} Sep 07 10:26:14.278798 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:14.278775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" event={"ID":"7da52000-a13f-44f3-aa14-7f10d4b44089","Type":"ContainerStarted","Data":"8c08321614fefda521dfd4bceb6c5ca140aab4573dd3fdc81f6e536ade987ff4"} Sep 07 10:26:14.278905 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:14.278831 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" event={"ID":"7da52000-a13f-44f3-aa14-7f10d4b44089","Type":"ContainerStarted","Data":"610d8304697d24a0e90aa0fa88fc823cfc1991778ec8750d9a44de57d2d72253"} Sep 07 10:26:14.292258 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:14.292209 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" podStartSLOduration=2.292189743 podStartE2EDuration="2.292189743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:14.291843112 +0000 UTC m=+896.278123446" watchObservedRunningTime="2026-09-07 10:26:14.292189743 +0000 UTC m=+896.278470075" Sep 07 10:26:14.306351 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:14.306260 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" podStartSLOduration=1.306245878 podStartE2EDuration="1.306245878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:14.304610465 +0000 UTC m=+896.290890799" watchObservedRunningTime="2026-09-07 10:26:14.306245878 +0000 UTC m=+896.292526212" Sep 07 10:26:14.309361 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:14.309330 2565 patch_prober.go:28] interesting pod/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 07 10:26:14.309481 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:14.309386 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 07 10:26:15.289420 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:15.289386 2565 generic.go:356] "Generic (PLEG): container finished" podID="7da52000-a13f-44f3-aa14-7f10d4b44089" containerID="8c08321614fefda521dfd4bceb6c5ca140aab4573dd3fdc81f6e536ade987ff4" exitCode=0 Sep 07 10:26:15.289831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:15.289448 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" event={"ID":"7da52000-a13f-44f3-aa14-7f10d4b44089","Type":"ContainerDied","Data":"8c08321614fefda521dfd4bceb6c5ca140aab4573dd3fdc81f6e536ade987ff4"} Sep 07 10:26:15.291067 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:15.291041 2565 generic.go:356] "Generic (PLEG): container finished" podID="fcc59765-bbdd-48aa-9bc9-5c541a4987d4" containerID="a8905c5b2bef80dc2c6230ffc9560360a7126a352f7d8c0898504fe1eea68c46" exitCode=0 Sep 07 10:26:15.291147 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:15.291100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" event={"ID":"fcc59765-bbdd-48aa-9bc9-5c541a4987d4","Type":"ContainerDied","Data":"a8905c5b2bef80dc2c6230ffc9560360a7126a352f7d8c0898504fe1eea68c46"} Sep 07 10:26:15.309357 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:15.309325 2565 patch_prober.go:28] interesting pod/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 07 10:26:15.309568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:15.309376 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 07 10:26:16.309315 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.309285 2565 patch_prober.go:28] interesting pod/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 07 10:26:16.309775 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.309346 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 07 10:26:16.479054 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.479030 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" Sep 07 10:26:16.513228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.513204 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" Sep 07 10:26:16.588256 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.588184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da52000-a13f-44f3-aa14-7f10d4b44089-maas-api-service-ca\") pod \"7da52000-a13f-44f3-aa14-7f10d4b44089\" (UID: \"7da52000-a13f-44f3-aa14-7f10d4b44089\") " Sep 07 10:26:16.588256 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.588236 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc59765-bbdd-48aa-9bc9-5c541a4987d4-maas-api-service-ca\") pod \"fcc59765-bbdd-48aa-9bc9-5c541a4987d4\" (UID: \"fcc59765-bbdd-48aa-9bc9-5c541a4987d4\") " Sep 07 10:26:16.588467 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.588445 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da52000-a13f-44f3-aa14-7f10d4b44089-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7da52000-a13f-44f3-aa14-7f10d4b44089" (UID: "7da52000-a13f-44f3-aa14-7f10d4b44089"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:26:16.588542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.588530 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da52000-a13f-44f3-aa14-7f10d4b44089-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:16.588576 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.588539 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc59765-bbdd-48aa-9bc9-5c541a4987d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcc59765-bbdd-48aa-9bc9-5c541a4987d4" (UID: "fcc59765-bbdd-48aa-9bc9-5c541a4987d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:26:16.689791 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:16.689757 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc59765-bbdd-48aa-9bc9-5c541a4987d4-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:17.099577 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.099548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:26:17.302787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.302763 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" Sep 07 10:26:17.302999 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.302789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x" event={"ID":"fcc59765-bbdd-48aa-9bc9-5c541a4987d4","Type":"ContainerDied","Data":"bf53d7ad21b31affc2204d9681a67491132bb45b6924578c2b3c82f5a2521160"} Sep 07 10:26:17.302999 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.302839 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf53d7ad21b31affc2204d9681a67491132bb45b6924578c2b3c82f5a2521160" Sep 07 10:26:17.304174 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.304139 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" Sep 07 10:26:17.304300 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.304138 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw" event={"ID":"7da52000-a13f-44f3-aa14-7f10d4b44089","Type":"ContainerDied","Data":"610d8304697d24a0e90aa0fa88fc823cfc1991778ec8750d9a44de57d2d72253"} Sep 07 10:26:17.304300 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.304292 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="610d8304697d24a0e90aa0fa88fc823cfc1991778ec8750d9a44de57d2d72253" Sep 07 10:26:17.308877 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.308847 2565 patch_prober.go:28] interesting pod/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 07 10:26:17.308996 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:17.308905 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 07 10:26:18.308892 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:18.308857 2565 patch_prober.go:28] interesting pod/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 07 10:26:18.309335 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:18.308909 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 07 10:26:18.525627 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:18.525602 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:26:18.526902 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:18.526880 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:26:19.312868 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:19.312841 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:26:19.313829 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:19.313811 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:26:19.862001 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:19.861950 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw"] Sep 07 10:26:19.864314 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:19.864290 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-31186eae-fpxlw"] Sep 07 10:26:20.544980 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:20.544949 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da52000-a13f-44f3-aa14-7f10d4b44089" path="/var/lib/kubelet/pods/7da52000-a13f-44f3-aa14-7f10d4b44089/volumes" Sep 07 10:26:20.922854 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:20.922760 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n"] Sep 07 10:26:21.327037 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.326997 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" containerID="cri-o://c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3" gracePeriod=30 Sep 07 10:26:21.608827 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.608737 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46"] Sep 07 10:26:21.609326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.609243 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da52000-a13f-44f3-aa14-7f10d4b44089" containerName="revoke-keys" Sep 07 10:26:21.609326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.609257 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da52000-a13f-44f3-aa14-7f10d4b44089" containerName="revoke-keys" Sep 07 10:26:21.609326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.609267 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc59765-bbdd-48aa-9bc9-5c541a4987d4" containerName="revoke-keys" Sep 07 10:26:21.609326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.609275 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc59765-bbdd-48aa-9bc9-5c541a4987d4" containerName="revoke-keys" Sep 07 10:26:21.609554 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.609345 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc59765-bbdd-48aa-9bc9-5c541a4987d4" containerName="revoke-keys" Sep 07 10:26:21.609554 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.609361 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da52000-a13f-44f3-aa14-7f10d4b44089" containerName="revoke-keys" Sep 07 10:26:21.614044 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.614022 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.616509 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.616488 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-39223a0b-openshift-default-dockercfg-zpc4g\"" Sep 07 10:26:21.627556 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.627527 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46"] Sep 07 10:26:21.746989 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.746953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-data\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.746989 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.746990 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-token\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.747295 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.747076 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-credential-socket\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.747295 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.747125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-certs\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.747295 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.747252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-envoy\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.747418 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.747297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232fe541-2e9f-43f2-aad0-c53063e85c86-istiod-ca-cert\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.747418 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.747328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxfj\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-kube-api-access-nmxfj\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.747418 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.747351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-socket\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.747418 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.747382 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-podinfo\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848289 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-envoy\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848476 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232fe541-2e9f-43f2-aad0-c53063e85c86-istiod-ca-cert\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848476 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxfj\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-kube-api-access-nmxfj\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-socket\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848571 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-podinfo\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848652 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-data\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848652 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-token\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848762 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-credential-socket\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848762 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-certs\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.848980 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-data\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.849051 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.848995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232fe541-2e9f-43f2-aad0-c53063e85c86-istiod-ca-cert\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.849105 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.849064 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-socket\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.849217 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.849194 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-credential-socket\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.849359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.849340 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-certs\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.850791 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.850768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-envoy\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.851200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.851150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-podinfo\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.856322 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.856302 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-token\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.856443 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.856424 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxfj\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-kube-api-access-nmxfj\") pod \"e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:21.925943 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:21.925855 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:22.205510 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:22.205482 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46"] Sep 07 10:26:22.207372 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:26:22.207345 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232fe541_2e9f_43f2_aad0_c53063e85c86.slice/crio-41b013d80cefacdfb9fbb34c92e6dd44f736e33923679d80e8201f8d768f6212 WatchSource:0}: Error finding container 41b013d80cefacdfb9fbb34c92e6dd44f736e33923679d80e8201f8d768f6212: Status 404 returned error can't find the container with id 41b013d80cefacdfb9fbb34c92e6dd44f736e33923679d80e8201f8d768f6212 Sep 07 10:26:22.210108 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:22.210073 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:22.210195 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:22.210140 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:22.210240 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:22.210192 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:22.335832 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:22.335801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" event={"ID":"232fe541-2e9f-43f2-aad0-c53063e85c86","Type":"ContainerStarted","Data":"1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529"} Sep 07 10:26:22.336146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:22.335852 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" event={"ID":"232fe541-2e9f-43f2-aad0-c53063e85c86","Type":"ContainerStarted","Data":"41b013d80cefacdfb9fbb34c92e6dd44f736e33923679d80e8201f8d768f6212"} Sep 07 10:26:22.357126 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:22.357073 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" podStartSLOduration=1.3570555180000001 podStartE2EDuration="1.357055518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:22.354415185 +0000 UTC m=+904.340695520" watchObservedRunningTime="2026-09-07 10:26:22.357055518 +0000 UTC m=+904.343335848" Sep 07 10:26:22.926282 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:22.926233 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:23.673013 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:23.672978 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv"] Sep 07 10:26:23.673229 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:23.673206 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" podUID="d86c3d94-0aba-4947-8540-3e08818173eb" containerName="payload-pre-processing" containerID="cri-o://89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2" gracePeriod=30 Sep 07 10:26:23.883902 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:23.883862 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x"] Sep 07 10:26:23.886275 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:23.886249 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ff5470-8mz9x"] Sep 07 10:26:23.932613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:23.932540 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:23.941956 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:23.941847 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:26:24.087224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.087191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72b2h\" (UniqueName: \"kubernetes.io/projected/d86c3d94-0aba-4947-8540-3e08818173eb-kube-api-access-72b2h\") pod \"d86c3d94-0aba-4947-8540-3e08818173eb\" (UID: \"d86c3d94-0aba-4947-8540-3e08818173eb\") " Sep 07 10:26:24.087394 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.087293 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86c3d94-0aba-4947-8540-3e08818173eb-plugins-config-volume\") pod \"d86c3d94-0aba-4947-8540-3e08818173eb\" (UID: \"d86c3d94-0aba-4947-8540-3e08818173eb\") " Sep 07 10:26:24.087660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.087627 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86c3d94-0aba-4947-8540-3e08818173eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d86c3d94-0aba-4947-8540-3e08818173eb" (UID: "d86c3d94-0aba-4947-8540-3e08818173eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:26:24.089411 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.089387 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86c3d94-0aba-4947-8540-3e08818173eb-kube-api-access-72b2h" (OuterVolumeSpecName: "kube-api-access-72b2h") pod "d86c3d94-0aba-4947-8540-3e08818173eb" (UID: "d86c3d94-0aba-4947-8540-3e08818173eb"). InnerVolumeSpecName "kube-api-access-72b2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:26:24.188907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.188818 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86c3d94-0aba-4947-8540-3e08818173eb-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:24.188907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.188851 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72b2h\" (UniqueName: \"kubernetes.io/projected/d86c3d94-0aba-4947-8540-3e08818173eb-kube-api-access-72b2h\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:24.350777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.350741 2565 generic.go:356] "Generic (PLEG): container finished" podID="d86c3d94-0aba-4947-8540-3e08818173eb" containerID="89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2" exitCode=0 Sep 07 10:26:24.350984 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.350794 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" Sep 07 10:26:24.350984 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.350827 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" event={"ID":"d86c3d94-0aba-4947-8540-3e08818173eb","Type":"ContainerDied","Data":"89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2"} Sep 07 10:26:24.350984 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.350877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv" event={"ID":"d86c3d94-0aba-4947-8540-3e08818173eb","Type":"ContainerDied","Data":"c99b102e9e0ed17f857e28a09c6de2e7b055785612ad2acb145977a5617b2fc8"} Sep 07 10:26:24.350984 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.350899 2565 scope.go:117] "RemoveContainer" containerID="89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2" Sep 07 10:26:24.351304 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.351287 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:24.352373 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.352358 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:24.360709 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.360693 2565 scope.go:117] "RemoveContainer" containerID="89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2" Sep 07 10:26:24.361007 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:26:24.360986 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2\": container with ID starting with 89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2 not found: ID does not exist" containerID="89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2" Sep 07 10:26:24.361080 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.361022 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2"} err="failed to get container status \"89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2\": rpc error: code = NotFound desc = could not find container \"89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2\": container with ID starting with 89d207fa6ef876c7b464dbef4f449430a975771d8e79400e5d4f0deaef24f9b2 not found: ID does not exist" Sep 07 10:26:24.392728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.392354 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv"] Sep 07 10:26:24.397589 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.397562 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ff5470-59db8f65bb-svpzv"] Sep 07 10:26:24.544843 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.544812 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86c3d94-0aba-4947-8540-3e08818173eb" path="/var/lib/kubelet/pods/d86c3d94-0aba-4947-8540-3e08818173eb/volumes" Sep 07 10:26:24.545204 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:24.545191 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc59765-bbdd-48aa-9bc9-5c541a4987d4" path="/var/lib/kubelet/pods/fcc59765-bbdd-48aa-9bc9-5c541a4987d4/volumes" Sep 07 10:26:26.602497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.602437 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:26:26.717109 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717071 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82b3aeb-bd27-42d8-b602-296f533d0702-istiod-ca-cert\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717325 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-data\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717325 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717207 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-podinfo\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717325 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95qjs\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-kube-api-access-95qjs\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717442 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717345 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-credential-socket\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717442 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717368 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-socket\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717442 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717393 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-envoy\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-token\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717505 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-certs\") pod \"f82b3aeb-bd27-42d8-b602-296f533d0702\" (UID: \"f82b3aeb-bd27-42d8-b602-296f533d0702\") " Sep 07 10:26:26.717594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717513 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-data" (OuterVolumeSpecName: "istio-data") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:26.717594 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717574 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82b3aeb-bd27-42d8-b602-296f533d0702-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:26:26.717814 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717627 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:26.717915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717879 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:26.718056 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717903 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:26.718056 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717934 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:26.718056 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717949 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82b3aeb-bd27-42d8-b602-296f533d0702-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:26.718056 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.717965 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:26.719738 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.719717 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:26:26.719837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.719818 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-token" (OuterVolumeSpecName: "istio-token") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:26:26.720139 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.720116 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-kube-api-access-95qjs" (OuterVolumeSpecName: "kube-api-access-95qjs") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "kube-api-access-95qjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:26:26.720260 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.720136 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f82b3aeb-bd27-42d8-b602-296f533d0702" (UID: "f82b3aeb-bd27-42d8-b602-296f533d0702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:26.819398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.819360 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:26.819398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.819389 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:26.819398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.819398 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:26.819398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.819407 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82b3aeb-bd27-42d8-b602-296f533d0702-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:26.819734 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.819414 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82b3aeb-bd27-42d8-b602-296f533d0702-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:26.819734 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:26.819422 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95qjs\" (UniqueName: \"kubernetes.io/projected/f82b3aeb-bd27-42d8-b602-296f533d0702-kube-api-access-95qjs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:27.369371 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.369339 2565 generic.go:356] "Generic (PLEG): container finished" podID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerID="c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3" exitCode=0 Sep 07 10:26:27.369534 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.369388 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" Sep 07 10:26:27.369534 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.369416 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" event={"ID":"f82b3aeb-bd27-42d8-b602-296f533d0702","Type":"ContainerDied","Data":"c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3"} Sep 07 10:26:27.369534 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.369477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n" event={"ID":"f82b3aeb-bd27-42d8-b602-296f533d0702","Type":"ContainerDied","Data":"537d04f8e8f39e63d74125853c21a3b87043efca9a1a6c1655b8d5c9e91dc491"} Sep 07 10:26:27.369534 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.369501 2565 scope.go:117] "RemoveContainer" containerID="c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3" Sep 07 10:26:27.379243 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.379227 2565 scope.go:117] "RemoveContainer" containerID="65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845" Sep 07 10:26:27.387453 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.387377 2565 scope.go:117] "RemoveContainer" containerID="c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3" Sep 07 10:26:27.387804 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:26:27.387757 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3\": container with ID starting with c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3 not found: ID does not exist" containerID="c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3" Sep 07 10:26:27.387918 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.387793 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3"} err="failed to get container status \"c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3\": rpc error: code = NotFound desc = could not find container \"c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3\": container with ID starting with c15862e6ea6e8c28f531ee478c8b3e0bc2f122df3e8d712f9fc80d06fb60e1d3 not found: ID does not exist" Sep 07 10:26:27.387918 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.387816 2565 scope.go:117] "RemoveContainer" containerID="65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845" Sep 07 10:26:27.388146 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:26:27.388121 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845\": container with ID starting with 65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845 not found: ID does not exist" containerID="65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845" Sep 07 10:26:27.388252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.388175 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845"} err="failed to get container status \"65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845\": rpc error: code = NotFound desc = could not find container \"65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845\": container with ID starting with 65fbcf31f113d96f728d5922b77bb2ae955c3000c4af8af82fb40800f1a60845 not found: ID does not exist" Sep 07 10:26:27.390767 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.390749 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n"] Sep 07 10:26:27.393946 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.393924 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-31186eae-openshift-default-77df6ccc96-7jw5n"] Sep 07 10:26:27.693634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.693537 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" podUID="3e599d6a-8772-4d73-bfba-7c579832b0b1" containerName="main" containerID="cri-o://c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b" gracePeriod=45 Sep 07 10:26:27.949858 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:27.949797 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:26:28.134228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134192 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-dshm\") pod \"3e599d6a-8772-4d73-bfba-7c579832b0b1\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " Sep 07 10:26:28.134400 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134238 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e599d6a-8772-4d73-bfba-7c579832b0b1-tls-certs\") pod \"3e599d6a-8772-4d73-bfba-7c579832b0b1\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " Sep 07 10:26:28.134400 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w4hk\" (UniqueName: \"kubernetes.io/projected/3e599d6a-8772-4d73-bfba-7c579832b0b1-kube-api-access-5w4hk\") pod \"3e599d6a-8772-4d73-bfba-7c579832b0b1\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " Sep 07 10:26:28.134400 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134356 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-tmp-dir\") pod \"3e599d6a-8772-4d73-bfba-7c579832b0b1\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " Sep 07 10:26:28.134400 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134383 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-model-cache\") pod \"3e599d6a-8772-4d73-bfba-7c579832b0b1\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " Sep 07 10:26:28.134637 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134497 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-home\") pod \"3e599d6a-8772-4d73-bfba-7c579832b0b1\" (UID: \"3e599d6a-8772-4d73-bfba-7c579832b0b1\") " Sep 07 10:26:28.134704 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134659 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e599d6a-8772-4d73-bfba-7c579832b0b1" (UID: "3e599d6a-8772-4d73-bfba-7c579832b0b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:28.134760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134697 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e599d6a-8772-4d73-bfba-7c579832b0b1" (UID: "3e599d6a-8772-4d73-bfba-7c579832b0b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:28.134801 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134764 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-home" (OuterVolumeSpecName: "home") pod "3e599d6a-8772-4d73-bfba-7c579832b0b1" (UID: "3e599d6a-8772-4d73-bfba-7c579832b0b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:28.134907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134894 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:28.134953 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134910 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:28.134953 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.134918 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:28.136700 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.136682 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e599d6a-8772-4d73-bfba-7c579832b0b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e599d6a-8772-4d73-bfba-7c579832b0b1" (UID: "3e599d6a-8772-4d73-bfba-7c579832b0b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:26:28.137000 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.136978 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-dshm" (OuterVolumeSpecName: "dshm") pod "3e599d6a-8772-4d73-bfba-7c579832b0b1" (UID: "3e599d6a-8772-4d73-bfba-7c579832b0b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:28.137062 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.137014 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e599d6a-8772-4d73-bfba-7c579832b0b1-kube-api-access-5w4hk" (OuterVolumeSpecName: "kube-api-access-5w4hk") pod "3e599d6a-8772-4d73-bfba-7c579832b0b1" (UID: "3e599d6a-8772-4d73-bfba-7c579832b0b1"). InnerVolumeSpecName "kube-api-access-5w4hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:26:28.235885 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.235851 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e599d6a-8772-4d73-bfba-7c579832b0b1-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:28.235885 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.235881 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e599d6a-8772-4d73-bfba-7c579832b0b1-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:28.235885 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.235893 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4hk\" (UniqueName: \"kubernetes.io/projected/3e599d6a-8772-4d73-bfba-7c579832b0b1-kube-api-access-5w4hk\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:28.375977 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.375940 2565 generic.go:356] "Generic (PLEG): container finished" podID="3e599d6a-8772-4d73-bfba-7c579832b0b1" containerID="c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b" exitCode=0 Sep 07 10:26:28.375977 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.375977 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" Sep 07 10:26:28.376273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.376026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" event={"ID":"3e599d6a-8772-4d73-bfba-7c579832b0b1","Type":"ContainerDied","Data":"c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b"} Sep 07 10:26:28.376273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.376069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58" event={"ID":"3e599d6a-8772-4d73-bfba-7c579832b0b1","Type":"ContainerDied","Data":"e557d2ac663ecfd73d00a5c0809fbd9de3f1bec8c11ba795c7da16e12f16068c"} Sep 07 10:26:28.376273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.376091 2565 scope.go:117] "RemoveContainer" containerID="c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b" Sep 07 10:26:28.385506 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.385489 2565 scope.go:117] "RemoveContainer" containerID="c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b" Sep 07 10:26:28.385770 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:26:28.385753 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b\": container with ID starting with c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b not found: ID does not exist" containerID="c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b" Sep 07 10:26:28.385836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.385777 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b"} err="failed to get container status \"c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b\": rpc error: code = NotFound desc = could not find container \"c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b\": container with ID starting with c3d065a896f2eb6d8c8b98b752d0dcd36bf188c873fa80b6719c86c850c66d9b not found: ID does not exist" Sep 07 10:26:28.397738 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.397713 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58"] Sep 07 10:26:28.400885 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.400861 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-ff5470/autores-model-ff5470-kserve-5f84fc5c4c-4kd58"] Sep 07 10:26:28.550335 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.550252 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e599d6a-8772-4d73-bfba-7c579832b0b1" path="/var/lib/kubelet/pods/3e599d6a-8772-4d73-bfba-7c579832b0b1/volumes" Sep 07 10:26:28.550716 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:28.550697 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" path="/var/lib/kubelet/pods/f82b3aeb-bd27-42d8-b602-296f533d0702/volumes" Sep 07 10:26:30.304336 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:30.304298 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf"] Sep 07 10:26:30.304737 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:30.304601 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" containerID="cri-o://43a1e705502ae7f6777a1e503115b628c8e0cb39469317bd5c0054cd8feb684b" gracePeriod=30 Sep 07 10:26:31.042650 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.042600 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts"] Sep 07 10:26:31.043725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.043703 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" Sep 07 10:26:31.043892 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.043880 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" Sep 07 10:26:31.043991 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.043968 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" Sep 07 10:26:31.043991 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.043987 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" Sep 07 10:26:31.044197 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.044007 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e599d6a-8772-4d73-bfba-7c579832b0b1" containerName="main" Sep 07 10:26:31.044197 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.044016 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e599d6a-8772-4d73-bfba-7c579832b0b1" containerName="main" Sep 07 10:26:31.044197 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.044040 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86c3d94-0aba-4947-8540-3e08818173eb" containerName="payload-pre-processing" Sep 07 10:26:31.044197 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.044049 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86c3d94-0aba-4947-8540-3e08818173eb" containerName="payload-pre-processing" Sep 07 10:26:31.044197 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.044195 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" Sep 07 10:26:31.044466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.044206 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86c3d94-0aba-4947-8540-3e08818173eb" containerName="payload-pre-processing" Sep 07 10:26:31.044466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.044219 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e599d6a-8772-4d73-bfba-7c579832b0b1" containerName="main" Sep 07 10:26:31.044466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.044232 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f82b3aeb-bd27-42d8-b602-296f533d0702" containerName="istio-proxy" Sep 07 10:26:31.049786 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.049763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.052659 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.052623 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-81af70-openshift-default-dockercfg-lkgcw\"" Sep 07 10:26:31.060541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.060513 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts"] Sep 07 10:26:31.170139 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhbn\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-kube-api-access-mbhbn\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.170358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-certs\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.170358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istiod-ca-cert\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.170358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-data\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.170358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-credential-socket\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.170358 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-envoy\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.170561 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170498 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-socket\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.170561 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-podinfo\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.170636 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.170568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-token\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271309 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-envoy\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-socket\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-podinfo\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-token\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhbn\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-kube-api-access-mbhbn\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271745 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-certs\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271745 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istiod-ca-cert\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271745 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-data\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271745 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-credential-socket\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271943 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-socket\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.271943 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.271855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-certs\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.272046 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.272016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-credential-socket\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.272268 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.272136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-data\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.272359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.272342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istiod-ca-cert\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.273824 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.273796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-envoy\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.273937 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.273923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-podinfo\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.279890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.279870 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhbn\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-kube-api-access-mbhbn\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.279978 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.279958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-token\") pod \"e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.371146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.371045 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:31.648953 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.648925 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts"] Sep 07 10:26:31.650110 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:26:31.650080 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012241eb_9b25_4b5f_9d0d_24d8aeba8485.slice/crio-ca530d84484fe83f2e9f676ac9921163c8c7080887eeda067d7f9aabc5755c13 WatchSource:0}: Error finding container ca530d84484fe83f2e9f676ac9921163c8c7080887eeda067d7f9aabc5755c13: Status 404 returned error can't find the container with id ca530d84484fe83f2e9f676ac9921163c8c7080887eeda067d7f9aabc5755c13 Sep 07 10:26:31.652578 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.652538 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:31.652660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.652605 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:31.652660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:31.652631 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:32.403869 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:32.403832 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" event={"ID":"012241eb-9b25-4b5f-9d0d-24d8aeba8485","Type":"ContainerStarted","Data":"12b11f282f3a7abee261b3acd67ff92a32216f49956c04e284236d59e6d4dfe4"} Sep 07 10:26:32.404265 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:32.403890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" event={"ID":"012241eb-9b25-4b5f-9d0d-24d8aeba8485","Type":"ContainerStarted","Data":"ca530d84484fe83f2e9f676ac9921163c8c7080887eeda067d7f9aabc5755c13"} Sep 07 10:26:32.424784 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:32.424736 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" podStartSLOduration=1.424719407 podStartE2EDuration="1.424719407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:32.423822598 +0000 UTC m=+914.410102968" watchObservedRunningTime="2026-09-07 10:26:32.424719407 +0000 UTC m=+914.410999741" Sep 07 10:26:33.371759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:33.371715 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:33.376695 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:33.376670 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:33.410430 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:33.410395 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:33.411595 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:33.411576 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:35.427403 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.427367 2565 generic.go:356] "Generic (PLEG): container finished" podID="4576c52d-4883-44c1-b5a4-478251329113" containerID="43a1e705502ae7f6777a1e503115b628c8e0cb39469317bd5c0054cd8feb684b" exitCode=0 Sep 07 10:26:35.427873 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.427459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" event={"ID":"4576c52d-4883-44c1-b5a4-478251329113","Type":"ContainerDied","Data":"43a1e705502ae7f6777a1e503115b628c8e0cb39469317bd5c0054cd8feb684b"} Sep 07 10:26:35.427873 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.427542 2565 scope.go:117] "RemoveContainer" containerID="ab448cacd7d0f1d70dacd9cc313db0d3cc8d2f81ae8d926dac49d68301f9fd10" Sep 07 10:26:35.615031 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.615004 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:26:35.717497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4576c52d-4883-44c1-b5a4-478251329113-istiod-ca-cert\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.717687 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-data\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.717777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717754 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-credential-socket\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.717848 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717834 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-certs\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.717902 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717893 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4576c52d-4883-44c1-b5a4-478251329113-istio-podinfo\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.717956 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717907 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4576c52d-4883-44c1-b5a4-478251329113-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:26:35.717956 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717923 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-data" (OuterVolumeSpecName: "istio-data") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:35.717956 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717929 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smtmx\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-kube-api-access-smtmx\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.718113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.717973 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:35.718113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718006 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-envoy\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.718113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718044 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-socket\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.718113 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718085 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-istio-token\") pod \"4576c52d-4883-44c1-b5a4-478251329113\" (UID: \"4576c52d-4883-44c1-b5a4-478251329113\") " Sep 07 10:26:35.718355 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718085 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:35.718355 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718296 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:35.718528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718508 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:35.718528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718528 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:35.718647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718537 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:35.718647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718546 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:35.718647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.718553 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4576c52d-4883-44c1-b5a4-478251329113-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:35.720340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.720309 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:26:35.720340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.720329 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-istio-token" (OuterVolumeSpecName: "istio-token") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:26:35.720700 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.720674 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4576c52d-4883-44c1-b5a4-478251329113-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:26:35.720797 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.720679 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-kube-api-access-smtmx" (OuterVolumeSpecName: "kube-api-access-smtmx") pod "4576c52d-4883-44c1-b5a4-478251329113" (UID: "4576c52d-4883-44c1-b5a4-478251329113"). InnerVolumeSpecName "kube-api-access-smtmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:26:35.819247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.819218 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4576c52d-4883-44c1-b5a4-478251329113-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:35.819247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.819245 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smtmx\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-kube-api-access-smtmx\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:35.819247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.819255 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4576c52d-4883-44c1-b5a4-478251329113-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:35.819469 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:35.819263 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4576c52d-4883-44c1-b5a4-478251329113-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:36.436763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:36.436716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" event={"ID":"4576c52d-4883-44c1-b5a4-478251329113","Type":"ContainerDied","Data":"0f20279308a62770b5d1202f454ec4048279a152ecbfa914c773b9462f2b2524"} Sep 07 10:26:36.437299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:36.436794 2565 scope.go:117] "RemoveContainer" containerID="43a1e705502ae7f6777a1e503115b628c8e0cb39469317bd5c0054cd8feb684b" Sep 07 10:26:36.437299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:36.436911 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf" Sep 07 10:26:36.468793 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:36.468757 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf"] Sep 07 10:26:36.470792 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:36.470772 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ff5470-openshift-default-76766d54bb-v77jf"] Sep 07 10:26:36.556106 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:36.556065 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4576c52d-4883-44c1-b5a4-478251329113" path="/var/lib/kubelet/pods/4576c52d-4883-44c1-b5a4-478251329113/volumes" Sep 07 10:26:40.074058 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.074024 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn"] Sep 07 10:26:40.074693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.074458 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" Sep 07 10:26:40.074693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.074471 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" Sep 07 10:26:40.074693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.074485 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" Sep 07 10:26:40.074693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.074490 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" Sep 07 10:26:40.074693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.074567 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" Sep 07 10:26:40.074693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.074574 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4576c52d-4883-44c1-b5a4-478251329113" containerName="istio-proxy" Sep 07 10:26:40.078934 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.078914 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.082913 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.082876 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f17c1f\"/\"default-dockercfg-852bn\"" Sep 07 10:26:40.083048 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.082876 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f17c1f\"/\"openshift-service-ca.crt\"" Sep 07 10:26:40.083048 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.082879 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f17c1f\"/\"kube-root-ca.crt\"" Sep 07 10:26:40.083188 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.082879 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f17c1f\"/\"test-model-f17c1f-kserve-self-signed-certs\"" Sep 07 10:26:40.087439 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.087416 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn"] Sep 07 10:26:40.172001 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.171957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6163f641-f733-4060-809f-5d2e6dfbb8e9-tls-certs\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.172181 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.172095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-tmp-dir\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.172181 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.172125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-home\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.172289 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.172152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-dshm\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.172289 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.172240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jlkv\" (UniqueName: \"kubernetes.io/projected/6163f641-f733-4060-809f-5d2e6dfbb8e9-kube-api-access-9jlkv\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.172364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.172314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-model-cache\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.274551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.273482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6163f641-f733-4060-809f-5d2e6dfbb8e9-tls-certs\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.274551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.273555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-tmp-dir\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.274551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.273583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-home\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.274551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.273610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-dshm\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.274551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.273669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jlkv\" (UniqueName: \"kubernetes.io/projected/6163f641-f733-4060-809f-5d2e6dfbb8e9-kube-api-access-9jlkv\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.274551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.273736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-model-cache\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.274551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.274212 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-model-cache\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.275279 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.274668 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-home\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.275279 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.274923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-tmp-dir\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.277321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.277294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6163f641-f733-4060-809f-5d2e6dfbb8e9-tls-certs\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.277633 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.277606 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-dshm\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.283716 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.283685 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jlkv\" (UniqueName: \"kubernetes.io/projected/6163f641-f733-4060-809f-5d2e6dfbb8e9-kube-api-access-9jlkv\") pod \"test-model-f17c1f-kserve-b7cf4d48f-77njn\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.390886 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.390776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:40.393354 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.393322 2565 patch_prober.go:28] interesting pod/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:26:40.393500 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.393394 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:26:40.713737 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:40.713702 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn"] Sep 07 10:26:40.715654 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:26:40.715626 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6163f641_f733_4060_809f_5d2e6dfbb8e9.slice/crio-9bc8e2fbe278144b1adf9b5bacd512cf47e5b3a581a85c61709de4e7729f087b WatchSource:0}: Error finding container 9bc8e2fbe278144b1adf9b5bacd512cf47e5b3a581a85c61709de4e7729f087b: Status 404 returned error can't find the container with id 9bc8e2fbe278144b1adf9b5bacd512cf47e5b3a581a85c61709de4e7729f087b Sep 07 10:26:41.485608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:41.485566 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" event={"ID":"6163f641-f733-4060-809f-5d2e6dfbb8e9","Type":"ContainerStarted","Data":"6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2"} Sep 07 10:26:41.485990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:41.485625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" event={"ID":"6163f641-f733-4060-809f-5d2e6dfbb8e9","Type":"ContainerStarted","Data":"9bc8e2fbe278144b1adf9b5bacd512cf47e5b3a581a85c61709de4e7729f087b"} Sep 07 10:26:41.506917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:41.506859 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" podStartSLOduration=1.5068382219999998 podStartE2EDuration="1.506838222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:41.503045126 +0000 UTC m=+923.489325459" watchObservedRunningTime="2026-09-07 10:26:41.506838222 +0000 UTC m=+923.493118557" Sep 07 10:26:43.498829 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:43.498796 2565 generic.go:356] "Generic (PLEG): container finished" podID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerID="1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529" exitCode=0 Sep 07 10:26:43.499296 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:43.498871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" event={"ID":"232fe541-2e9f-43f2-aad0-c53063e85c86","Type":"ContainerDied","Data":"1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529"} Sep 07 10:26:43.499381 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:43.499366 2565 scope.go:117] "RemoveContainer" containerID="1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529" Sep 07 10:26:43.500557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:43.500529 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:43.500630 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:43.500597 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:43.500671 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:43.500628 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:44.506725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:44.506693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" event={"ID":"232fe541-2e9f-43f2-aad0-c53063e85c86","Type":"ContainerStarted","Data":"bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b"} Sep 07 10:26:44.927107 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:44.927005 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:44.927310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:44.927238 2565 patch_prober.go:28] interesting pod/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 07 10:26:44.927310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:44.927284 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 07 10:26:45.926474 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:45.926439 2565 patch_prober.go:28] interesting pod/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 07 10:26:45.926860 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:45.926500 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 07 10:26:46.927317 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:46.927278 2565 patch_prober.go:28] interesting pod/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 07 10:26:46.927712 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:46.927351 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 07 10:26:47.926926 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:47.926880 2565 patch_prober.go:28] interesting pod/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 07 10:26:47.927146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:47.926958 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 07 10:26:48.519056 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.519022 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp"] Sep 07 10:26:48.522909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.522889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" Sep 07 10:26:48.525526 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.525504 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:26:48.525688 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.525504 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:26:48.543692 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.543660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp"] Sep 07 10:26:48.670337 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.670298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27def301-59b2-44da-8a66-6f6d1bc60c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39223a0b-lqztp\" (UID: \"27def301-59b2-44da-8a66-6f6d1bc60c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" Sep 07 10:26:48.771271 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.771188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27def301-59b2-44da-8a66-6f6d1bc60c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39223a0b-lqztp\" (UID: \"27def301-59b2-44da-8a66-6f6d1bc60c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" Sep 07 10:26:48.771909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.771881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27def301-59b2-44da-8a66-6f6d1bc60c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39223a0b-lqztp\" (UID: \"27def301-59b2-44da-8a66-6f6d1bc60c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" Sep 07 10:26:48.868813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.868776 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" Sep 07 10:26:48.926843 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.926811 2565 patch_prober.go:28] interesting pod/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 07 10:26:48.927042 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:48.927014 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 07 10:26:49.145013 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:49.144912 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp"] Sep 07 10:26:49.163899 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:26:49.163867 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27def301_59b2_44da_8a66_6f6d1bc60c4b.slice/crio-b17963c2c93ed355deaf5f52a9169cb3393c2716ad79a877727a412be0ffde27 WatchSource:0}: Error finding container b17963c2c93ed355deaf5f52a9169cb3393c2716ad79a877727a412be0ffde27: Status 404 returned error can't find the container with id b17963c2c93ed355deaf5f52a9169cb3393c2716ad79a877727a412be0ffde27 Sep 07 10:26:49.452499 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:49.452355 2565 patch_prober.go:28] interesting pod/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:26:49.452499 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:49.452457 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:26:49.539097 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:49.539060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" event={"ID":"27def301-59b2-44da-8a66-6f6d1bc60c4b","Type":"ContainerStarted","Data":"b17963c2c93ed355deaf5f52a9169cb3393c2716ad79a877727a412be0ffde27"} Sep 07 10:26:49.950566 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:49.950532 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:49.950930 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:49.950907 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:49.951862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:49.951837 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:26:50.394530 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.394135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:50.394530 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.394211 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:50.406734 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.406706 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:50.549726 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.549689 2565 generic.go:356] "Generic (PLEG): container finished" podID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerID="12b11f282f3a7abee261b3acd67ff92a32216f49956c04e284236d59e6d4dfe4" exitCode=0 Sep 07 10:26:50.550277 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.549914 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" event={"ID":"012241eb-9b25-4b5f-9d0d-24d8aeba8485","Type":"ContainerDied","Data":"12b11f282f3a7abee261b3acd67ff92a32216f49956c04e284236d59e6d4dfe4"} Sep 07 10:26:50.550613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.550588 2565 scope.go:117] "RemoveContainer" containerID="12b11f282f3a7abee261b3acd67ff92a32216f49956c04e284236d59e6d4dfe4" Sep 07 10:26:50.554752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.553083 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:50.554752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.553112 2565 generic.go:356] "Generic (PLEG): container finished" podID="27def301-59b2-44da-8a66-6f6d1bc60c4b" containerID="f809bbc3f775a56279f3a6c896452eeddeead5912c32e66d0eff55835baf4328" exitCode=0 Sep 07 10:26:50.554752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.553228 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:50.554752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.553283 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:26:50.554752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.554057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" event={"ID":"27def301-59b2-44da-8a66-6f6d1bc60c4b","Type":"ContainerDied","Data":"f809bbc3f775a56279f3a6c896452eeddeead5912c32e66d0eff55835baf4328"} Sep 07 10:26:50.575750 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:50.575719 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:26:51.563069 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:51.563032 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" event={"ID":"012241eb-9b25-4b5f-9d0d-24d8aeba8485","Type":"ContainerStarted","Data":"4ca93df959e0261b3ebb33601f427f0c4c6955e42074255b86c36323d12611fb"} Sep 07 10:26:51.718525 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:51.718498 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" Sep 07 10:26:51.805564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:51.805533 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27def301-59b2-44da-8a66-6f6d1bc60c4b-maas-api-service-ca\") pod \"27def301-59b2-44da-8a66-6f6d1bc60c4b\" (UID: \"27def301-59b2-44da-8a66-6f6d1bc60c4b\") " Sep 07 10:26:51.805888 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:51.805862 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27def301-59b2-44da-8a66-6f6d1bc60c4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27def301-59b2-44da-8a66-6f6d1bc60c4b" (UID: "27def301-59b2-44da-8a66-6f6d1bc60c4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:26:51.906746 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:51.906661 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27def301-59b2-44da-8a66-6f6d1bc60c4b-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:26:52.372001 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:52.371963 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:52.372229 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:52.372203 2565 patch_prober.go:28] interesting pod/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 07 10:26:52.372283 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:52.372258 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 07 10:26:52.569839 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:52.569816 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" Sep 07 10:26:52.570227 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:52.569852 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp" event={"ID":"27def301-59b2-44da-8a66-6f6d1bc60c4b","Type":"ContainerDied","Data":"b17963c2c93ed355deaf5f52a9169cb3393c2716ad79a877727a412be0ffde27"} Sep 07 10:26:52.570227 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:52.569895 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17963c2c93ed355deaf5f52a9169cb3393c2716ad79a877727a412be0ffde27" Sep 07 10:26:53.372011 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:53.371981 2565 patch_prober.go:28] interesting pod/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 07 10:26:53.372239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:53.372038 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 07 10:26:54.371703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:54.371671 2565 patch_prober.go:28] interesting pod/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 07 10:26:54.372091 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:54.371726 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 07 10:26:55.371787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:55.371747 2565 patch_prober.go:28] interesting pod/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 07 10:26:55.372214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:55.371820 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 07 10:26:55.761096 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:55.761055 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp"] Sep 07 10:26:55.763889 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:55.763859 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39223a0b-lqztp"] Sep 07 10:26:55.903185 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:55.902822 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46"] Sep 07 10:26:55.903382 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:55.903227 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" containerID="cri-o://bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b" gracePeriod=30 Sep 07 10:26:56.371939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:56.371898 2565 patch_prober.go:28] interesting pod/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 07 10:26:56.372370 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:56.371957 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 07 10:26:56.548823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:56.548734 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27def301-59b2-44da-8a66-6f6d1bc60c4b" path="/var/lib/kubelet/pods/27def301-59b2-44da-8a66-6f6d1bc60c4b/volumes" Sep 07 10:26:57.375656 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:57.375622 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:57.376045 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:57.376000 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:26:57.376769 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:26:57.376749 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:27:01.170028 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.170001 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:27:01.315100 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315066 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-podinfo\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315100 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315107 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-certs\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315391 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315132 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-envoy\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315391 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315206 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmxfj\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-kube-api-access-nmxfj\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315391 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315255 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-data\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315391 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315297 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-socket\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315391 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315348 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232fe541-2e9f-43f2-aad0-c53063e85c86-istiod-ca-cert\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315391 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315387 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-credential-socket\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315706 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315411 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-token\") pod \"232fe541-2e9f-43f2-aad0-c53063e85c86\" (UID: \"232fe541-2e9f-43f2-aad0-c53063e85c86\") " Sep 07 10:27:01.315706 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315418 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:01.315706 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315590 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:01.315706 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315665 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:01.315902 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315753 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.315902 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315765 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.315902 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315773 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.315902 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315828 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232fe541-2e9f-43f2-aad0-c53063e85c86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:27:01.315902 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.315891 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-data" (OuterVolumeSpecName: "istio-data") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:01.317678 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.317649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-kube-api-access-nmxfj" (OuterVolumeSpecName: "kube-api-access-nmxfj") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "kube-api-access-nmxfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:27:01.317794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.317690 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:01.317794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.317696 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-token" (OuterVolumeSpecName: "istio-token") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:27:01.317794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.317747 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "232fe541-2e9f-43f2-aad0-c53063e85c86" (UID: "232fe541-2e9f-43f2-aad0-c53063e85c86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:27:01.416894 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.416852 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.416894 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.416890 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232fe541-2e9f-43f2-aad0-c53063e85c86-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.416894 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.416902 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.417195 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.416912 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.417195 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.416923 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232fe541-2e9f-43f2-aad0-c53063e85c86-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.417195 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.416936 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxfj\" (UniqueName: \"kubernetes.io/projected/232fe541-2e9f-43f2-aad0-c53063e85c86-kube-api-access-nmxfj\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:01.627064 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.626969 2565 generic.go:356] "Generic (PLEG): container finished" podID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerID="bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b" exitCode=0 Sep 07 10:27:01.627064 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.627029 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" Sep 07 10:27:01.627064 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.627049 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" event={"ID":"232fe541-2e9f-43f2-aad0-c53063e85c86","Type":"ContainerDied","Data":"bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b"} Sep 07 10:27:01.627363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.627129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46" event={"ID":"232fe541-2e9f-43f2-aad0-c53063e85c86","Type":"ContainerDied","Data":"41b013d80cefacdfb9fbb34c92e6dd44f736e33923679d80e8201f8d768f6212"} Sep 07 10:27:01.627363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.627154 2565 scope.go:117] "RemoveContainer" containerID="bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b" Sep 07 10:27:01.637272 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.637252 2565 scope.go:117] "RemoveContainer" containerID="1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529" Sep 07 10:27:01.646243 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.646224 2565 scope.go:117] "RemoveContainer" containerID="bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b" Sep 07 10:27:01.646565 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:27:01.646544 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b\": container with ID starting with bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b not found: ID does not exist" containerID="bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b" Sep 07 10:27:01.646619 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.646576 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b"} err="failed to get container status \"bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b\": rpc error: code = NotFound desc = could not find container \"bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b\": container with ID starting with bf01516772772776cafb6929e35f94b58aa0a0097802133c0fe37e7540b4e09b not found: ID does not exist" Sep 07 10:27:01.646619 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.646594 2565 scope.go:117] "RemoveContainer" containerID="1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529" Sep 07 10:27:01.646856 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:27:01.646839 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529\": container with ID starting with 1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529 not found: ID does not exist" containerID="1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529" Sep 07 10:27:01.646899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.646862 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529"} err="failed to get container status \"1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529\": rpc error: code = NotFound desc = could not find container \"1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529\": container with ID starting with 1dfdb2be3af59d10ddbc87b3b8f7dadf30c73efee04c0acb578844ce924a0529 not found: ID does not exist" Sep 07 10:27:01.651968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.651944 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46"] Sep 07 10:27:01.664672 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:01.664641 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-39223a0b-openshift-default-c87cc77cb-jcn46"] Sep 07 10:27:02.544340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:02.544303 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" path="/var/lib/kubelet/pods/232fe541-2e9f-43f2-aad0-c53063e85c86/volumes" Sep 07 10:27:27.491511 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.491477 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4"] Sep 07 10:27:27.491975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.491925 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" Sep 07 10:27:27.491975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.491937 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" Sep 07 10:27:27.491975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.491946 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" Sep 07 10:27:27.491975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.491951 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" Sep 07 10:27:27.491975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.491958 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27def301-59b2-44da-8a66-6f6d1bc60c4b" containerName="revoke-keys" Sep 07 10:27:27.491975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.491963 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="27def301-59b2-44da-8a66-6f6d1bc60c4b" containerName="revoke-keys" Sep 07 10:27:27.492177 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.492044 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" Sep 07 10:27:27.492177 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.492059 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="27def301-59b2-44da-8a66-6f6d1bc60c4b" containerName="revoke-keys" Sep 07 10:27:27.496862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.496845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" Sep 07 10:27:27.501332 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.501237 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:27:27.501332 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.501237 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:27:27.503541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.503515 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4"] Sep 07 10:27:27.585150 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.585112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8afdb94-6385-435b-bbcd-408197bf2e69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-81af70-bvpd4\" (UID: \"f8afdb94-6385-435b-bbcd-408197bf2e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" Sep 07 10:27:27.686542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.686501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8afdb94-6385-435b-bbcd-408197bf2e69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-81af70-bvpd4\" (UID: \"f8afdb94-6385-435b-bbcd-408197bf2e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" Sep 07 10:27:27.687208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.687189 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8afdb94-6385-435b-bbcd-408197bf2e69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-81af70-bvpd4\" (UID: \"f8afdb94-6385-435b-bbcd-408197bf2e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" Sep 07 10:27:27.810397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:27.810301 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" Sep 07 10:27:28.084145 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:28.084120 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4"] Sep 07 10:27:28.087120 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:27:28.087083 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8afdb94_6385_435b_bbcd_408197bf2e69.slice/crio-789f8a911770f939fe8234cb895edfdca7f51c9967912531b59b451d1896644f WatchSource:0}: Error finding container 789f8a911770f939fe8234cb895edfdca7f51c9967912531b59b451d1896644f: Status 404 returned error can't find the container with id 789f8a911770f939fe8234cb895edfdca7f51c9967912531b59b451d1896644f Sep 07 10:27:28.778299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:28.778271 2565 generic.go:356] "Generic (PLEG): container finished" podID="f8afdb94-6385-435b-bbcd-408197bf2e69" containerID="5e90cc46c4de5195f5f3be39fe2a33bab4f00cbae418117949dd7336bf890de6" exitCode=0 Sep 07 10:27:28.778687 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:28.778347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" event={"ID":"f8afdb94-6385-435b-bbcd-408197bf2e69","Type":"ContainerDied","Data":"5e90cc46c4de5195f5f3be39fe2a33bab4f00cbae418117949dd7336bf890de6"} Sep 07 10:27:28.778687 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:28.778398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" event={"ID":"f8afdb94-6385-435b-bbcd-408197bf2e69","Type":"ContainerStarted","Data":"789f8a911770f939fe8234cb895edfdca7f51c9967912531b59b451d1896644f"} Sep 07 10:27:29.926546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:29.926520 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" Sep 07 10:27:30.011438 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:30.011401 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8afdb94-6385-435b-bbcd-408197bf2e69-maas-api-service-ca\") pod \"f8afdb94-6385-435b-bbcd-408197bf2e69\" (UID: \"f8afdb94-6385-435b-bbcd-408197bf2e69\") " Sep 07 10:27:30.011798 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:30.011774 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8afdb94-6385-435b-bbcd-408197bf2e69-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8afdb94-6385-435b-bbcd-408197bf2e69" (UID: "f8afdb94-6385-435b-bbcd-408197bf2e69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:27:30.113079 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:30.112974 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8afdb94-6385-435b-bbcd-408197bf2e69-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:30.795861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:30.794813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" event={"ID":"f8afdb94-6385-435b-bbcd-408197bf2e69","Type":"ContainerDied","Data":"789f8a911770f939fe8234cb895edfdca7f51c9967912531b59b451d1896644f"} Sep 07 10:27:30.795861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:30.794911 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789f8a911770f939fe8234cb895edfdca7f51c9967912531b59b451d1896644f" Sep 07 10:27:30.795861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:30.794983 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4" Sep 07 10:27:38.314599 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:38.314556 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4"] Sep 07 10:27:38.317656 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:38.317621 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-81af70-bvpd4"] Sep 07 10:27:38.545776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:38.545742 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8afdb94-6385-435b-bbcd-408197bf2e69" path="/var/lib/kubelet/pods/f8afdb94-6385-435b-bbcd-408197bf2e69/volumes" Sep 07 10:27:44.723491 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:44.723417 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts"] Sep 07 10:27:44.723911 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:44.723678 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" containerID="cri-o://4ca93df959e0261b3ebb33601f427f0c4c6955e42074255b86c36323d12611fb" gracePeriod=30 Sep 07 10:27:45.427022 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.426987 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb"] Sep 07 10:27:45.427618 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.427578 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8afdb94-6385-435b-bbcd-408197bf2e69" containerName="revoke-keys" Sep 07 10:27:45.427618 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.427604 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8afdb94-6385-435b-bbcd-408197bf2e69" containerName="revoke-keys" Sep 07 10:27:45.427797 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.427703 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8afdb94-6385-435b-bbcd-408197bf2e69" containerName="revoke-keys" Sep 07 10:27:45.427797 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.427718 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="232fe541-2e9f-43f2-aad0-c53063e85c86" containerName="istio-proxy" Sep 07 10:27:45.431404 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.431377 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.433876 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.433859 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-2d94a5-openshift-default-dockercfg-tnzqq\"" Sep 07 10:27:45.442028 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.442007 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb"] Sep 07 10:27:45.471743 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.471712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-certs\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.471915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.471749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-envoy\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.471915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.471767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-credential-socket\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.472034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.471912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-data\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.472034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.471948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-podinfo\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.472034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.471975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-socket\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.472034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.472029 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-token\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.472185 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.472054 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqqhd\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-kube-api-access-kqqhd\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.472185 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.472093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67574581-cb8a-421b-86a3-7998d17dfe6a-istiod-ca-cert\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.572905 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.572861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-data\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.572905 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.572905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-podinfo\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.572929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-socket\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-token\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqqhd\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-kube-api-access-kqqhd\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67574581-cb8a-421b-86a3-7998d17dfe6a-istiod-ca-cert\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-certs\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-envoy\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-credential-socket\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-data\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573923 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573897 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-credential-socket\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.573923 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573916 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-socket\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.574127 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.573984 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-certs\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.574443 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.574373 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67574581-cb8a-421b-86a3-7998d17dfe6a-istiod-ca-cert\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.575739 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.575719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-podinfo\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.575994 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.575974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-envoy\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.581635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.581612 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-token\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.581999 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.581976 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqqhd\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-kube-api-access-kqqhd\") pod \"e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:45.743699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:45.743662 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:46.021203 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:46.021173 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb"] Sep 07 10:27:46.022129 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:27:46.022100 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67574581_cb8a_421b_86a3_7998d17dfe6a.slice/crio-01723246cc1bbae34a1190adcb6127f50693bc58bb85fc7f832013eab635037e WatchSource:0}: Error finding container 01723246cc1bbae34a1190adcb6127f50693bc58bb85fc7f832013eab635037e: Status 404 returned error can't find the container with id 01723246cc1bbae34a1190adcb6127f50693bc58bb85fc7f832013eab635037e Sep 07 10:27:46.024643 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:46.024597 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:27:46.024757 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:46.024699 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:27:46.024797 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:46.024777 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:27:46.885392 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:46.885354 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" event={"ID":"67574581-cb8a-421b-86a3-7998d17dfe6a","Type":"ContainerStarted","Data":"4d5bd0d11783f43566d801d2d7b944459a064dedb1e5420849803987d4a32aaf"} Sep 07 10:27:46.885809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:46.885406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" event={"ID":"67574581-cb8a-421b-86a3-7998d17dfe6a","Type":"ContainerStarted","Data":"01723246cc1bbae34a1190adcb6127f50693bc58bb85fc7f832013eab635037e"} Sep 07 10:27:46.911547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:46.911491 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" podStartSLOduration=1.911476822 podStartE2EDuration="1.911476822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:27:46.908323436 +0000 UTC m=+988.894603800" watchObservedRunningTime="2026-09-07 10:27:46.911476822 +0000 UTC m=+988.897757155" Sep 07 10:27:47.744348 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:47.744305 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:47.749443 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:47.749416 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:47.890863 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:47.890832 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:47.891867 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:47.891848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:27:49.908064 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:49.908034 2565 generic.go:356] "Generic (PLEG): container finished" podID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerID="4ca93df959e0261b3ebb33601f427f0c4c6955e42074255b86c36323d12611fb" exitCode=0 Sep 07 10:27:49.908546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:49.908113 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" event={"ID":"012241eb-9b25-4b5f-9d0d-24d8aeba8485","Type":"ContainerDied","Data":"4ca93df959e0261b3ebb33601f427f0c4c6955e42074255b86c36323d12611fb"} Sep 07 10:27:49.908546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:49.908208 2565 scope.go:117] "RemoveContainer" containerID="12b11f282f3a7abee261b3acd67ff92a32216f49956c04e284236d59e6d4dfe4" Sep 07 10:27:49.982287 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:49.982261 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:27:50.022289 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022201 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbhbn\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-kube-api-access-mbhbn\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022289 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022236 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-socket\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022289 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022258 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-data\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022289 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022280 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-envoy\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022625 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022305 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-credential-socket\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022625 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022327 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-token\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022625 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022358 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-podinfo\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022625 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022436 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-certs\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022625 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022470 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istiod-ca-cert\") pod \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\" (UID: \"012241eb-9b25-4b5f-9d0d-24d8aeba8485\") " Sep 07 10:27:50.022625 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022557 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:50.022943 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:27:50.022943 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022868 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-data" (OuterVolumeSpecName: "istio-data") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:50.022943 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.022879 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:50.023115 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.023090 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:50.023211 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.023134 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.023211 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.023154 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.023211 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.023192 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.023211 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.023205 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.025221 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.025192 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-token" (OuterVolumeSpecName: "istio-token") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:27:50.025575 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.025551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:27:50.025719 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.025701 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-kube-api-access-mbhbn" (OuterVolumeSpecName: "kube-api-access-mbhbn") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "kube-api-access-mbhbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:27:50.025768 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.025729 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "012241eb-9b25-4b5f-9d0d-24d8aeba8485" (UID: "012241eb-9b25-4b5f-9d0d-24d8aeba8485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:27:50.123874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.123835 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbhbn\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-kube-api-access-mbhbn\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.123874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.123867 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.123874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.123878 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012241eb-9b25-4b5f-9d0d-24d8aeba8485-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.123874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.123887 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.124147 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.123895 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012241eb-9b25-4b5f-9d0d-24d8aeba8485-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:27:50.914347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.914264 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" Sep 07 10:27:50.914347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.914305 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts" event={"ID":"012241eb-9b25-4b5f-9d0d-24d8aeba8485","Type":"ContainerDied","Data":"ca530d84484fe83f2e9f676ac9921163c8c7080887eeda067d7f9aabc5755c13"} Sep 07 10:27:50.914820 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.914363 2565 scope.go:117] "RemoveContainer" containerID="4ca93df959e0261b3ebb33601f427f0c4c6955e42074255b86c36323d12611fb" Sep 07 10:27:50.932655 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.932633 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts"] Sep 07 10:27:50.936546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:50.936525 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-81af70-openshift-default-586f5bfd6d-fnpts"] Sep 07 10:27:52.544293 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:27:52.544258 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" path="/var/lib/kubelet/pods/012241eb-9b25-4b5f-9d0d-24d8aeba8485/volumes" Sep 07 10:28:15.877624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:15.877589 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb"] Sep 07 10:28:15.878092 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:15.877893 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" podUID="67574581-cb8a-421b-86a3-7998d17dfe6a" containerName="istio-proxy" containerID="cri-o://4d5bd0d11783f43566d801d2d7b944459a064dedb1e5420849803987d4a32aaf" gracePeriod=30 Sep 07 10:28:16.780961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.780924 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn"] Sep 07 10:28:16.896425 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.896387 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz"] Sep 07 10:28:16.896797 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.896775 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" Sep 07 10:28:16.896797 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.896785 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" Sep 07 10:28:16.896876 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.896866 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" Sep 07 10:28:16.896876 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.896874 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" Sep 07 10:28:16.900072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.900055 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" Sep 07 10:28:16.902776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.902756 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:28:16.902897 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.902787 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:28:16.907760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:16.907732 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz"] Sep 07 10:28:17.009933 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:17.009895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526e5d0a-09da-4d0a-afcc-2ca165fb2b17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz\" (UID: \"526e5d0a-09da-4d0a-afcc-2ca165fb2b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" Sep 07 10:28:17.110934 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:17.110849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526e5d0a-09da-4d0a-afcc-2ca165fb2b17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz\" (UID: \"526e5d0a-09da-4d0a-afcc-2ca165fb2b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" Sep 07 10:28:17.111481 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:17.111462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526e5d0a-09da-4d0a-afcc-2ca165fb2b17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz\" (UID: \"526e5d0a-09da-4d0a-afcc-2ca165fb2b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" Sep 07 10:28:17.214068 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:17.214035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" Sep 07 10:28:17.485472 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:17.485447 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz"] Sep 07 10:28:17.487826 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:28:17.487796 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526e5d0a_09da_4d0a_afcc_2ca165fb2b17.slice/crio-3e5c3bdc8f760a6cea0e8abe24cfeed0b725d4fe8d253705066b2fb422872183 WatchSource:0}: Error finding container 3e5c3bdc8f760a6cea0e8abe24cfeed0b725d4fe8d253705066b2fb422872183: Status 404 returned error can't find the container with id 3e5c3bdc8f760a6cea0e8abe24cfeed0b725d4fe8d253705066b2fb422872183 Sep 07 10:28:17.893355 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:17.893315 2565 patch_prober.go:28] interesting pod/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 10:28:17.893520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:17.893383 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" podUID="67574581-cb8a-421b-86a3-7998d17dfe6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:28:18.081721 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:18.081612 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" event={"ID":"526e5d0a-09da-4d0a-afcc-2ca165fb2b17","Type":"ContainerStarted","Data":"461ba9afb4dfc7b052a61fe4a265b7b91b485cd6384fc4c6497d261a83bf458b"} Sep 07 10:28:18.081721 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:18.081673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" event={"ID":"526e5d0a-09da-4d0a-afcc-2ca165fb2b17","Type":"ContainerStarted","Data":"3e5c3bdc8f760a6cea0e8abe24cfeed0b725d4fe8d253705066b2fb422872183"} Sep 07 10:28:19.088284 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:19.088254 2565 generic.go:356] "Generic (PLEG): container finished" podID="526e5d0a-09da-4d0a-afcc-2ca165fb2b17" containerID="461ba9afb4dfc7b052a61fe4a265b7b91b485cd6384fc4c6497d261a83bf458b" exitCode=0 Sep 07 10:28:19.088668 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:19.088349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" event={"ID":"526e5d0a-09da-4d0a-afcc-2ca165fb2b17","Type":"ContainerDied","Data":"461ba9afb4dfc7b052a61fe4a265b7b91b485cd6384fc4c6497d261a83bf458b"} Sep 07 10:28:20.232254 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:20.232230 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" Sep 07 10:28:20.341527 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:20.341496 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526e5d0a-09da-4d0a-afcc-2ca165fb2b17-maas-api-service-ca\") pod \"526e5d0a-09da-4d0a-afcc-2ca165fb2b17\" (UID: \"526e5d0a-09da-4d0a-afcc-2ca165fb2b17\") " Sep 07 10:28:20.341871 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:20.341848 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526e5d0a-09da-4d0a-afcc-2ca165fb2b17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "526e5d0a-09da-4d0a-afcc-2ca165fb2b17" (UID: "526e5d0a-09da-4d0a-afcc-2ca165fb2b17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:28:20.442451 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:20.442363 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526e5d0a-09da-4d0a-afcc-2ca165fb2b17-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.102959 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.102924 2565 generic.go:356] "Generic (PLEG): container finished" podID="67574581-cb8a-421b-86a3-7998d17dfe6a" containerID="4d5bd0d11783f43566d801d2d7b944459a064dedb1e5420849803987d4a32aaf" exitCode=0 Sep 07 10:28:21.103184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.102997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" event={"ID":"67574581-cb8a-421b-86a3-7998d17dfe6a","Type":"ContainerDied","Data":"4d5bd0d11783f43566d801d2d7b944459a064dedb1e5420849803987d4a32aaf"} Sep 07 10:28:21.104487 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.104465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" event={"ID":"526e5d0a-09da-4d0a-afcc-2ca165fb2b17","Type":"ContainerDied","Data":"3e5c3bdc8f760a6cea0e8abe24cfeed0b725d4fe8d253705066b2fb422872183"} Sep 07 10:28:21.104487 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.104484 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz" Sep 07 10:28:21.104628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.104506 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e5c3bdc8f760a6cea0e8abe24cfeed0b725d4fe8d253705066b2fb422872183" Sep 07 10:28:21.178400 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.178373 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:28:21.251717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.251688 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-podinfo\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.251734 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-socket\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.251785 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-credential-socket\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.251818 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-token\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.251860 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqqhd\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-kube-api-access-kqqhd\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.251935 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-envoy\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.251968 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67574581-cb8a-421b-86a3-7998d17dfe6a-istiod-ca-cert\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.251995 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-certs\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.252027 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:21.252228 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.252052 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-data\") pod \"67574581-cb8a-421b-86a3-7998d17dfe6a\" (UID: \"67574581-cb8a-421b-86a3-7998d17dfe6a\") " Sep 07 10:28:21.252710 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.252441 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.252710 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.252679 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-data" (OuterVolumeSpecName: "istio-data") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:21.252781 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.252721 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:21.252978 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.252878 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67574581-cb8a-421b-86a3-7998d17dfe6a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:28:21.253032 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.252964 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:21.255106 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.255067 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-token" (OuterVolumeSpecName: "istio-token") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:28:21.255106 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.255082 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-kube-api-access-kqqhd" (OuterVolumeSpecName: "kube-api-access-kqqhd") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "kube-api-access-kqqhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:28:21.255302 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.255126 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:28:21.255302 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.255265 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67574581-cb8a-421b-86a3-7998d17dfe6a" (UID: "67574581-cb8a-421b-86a3-7998d17dfe6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:21.353770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.353565 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.353770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.353603 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67574581-cb8a-421b-86a3-7998d17dfe6a-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.353770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.353617 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.353770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.353629 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.353770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.353646 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.353770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.353660 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67574581-cb8a-421b-86a3-7998d17dfe6a-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.353770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.353676 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:21.353770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:21.353689 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqqhd\" (UniqueName: \"kubernetes.io/projected/67574581-cb8a-421b-86a3-7998d17dfe6a-kube-api-access-kqqhd\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:22.111202 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:22.111174 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" Sep 07 10:28:22.111385 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:22.111203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb" event={"ID":"67574581-cb8a-421b-86a3-7998d17dfe6a","Type":"ContainerDied","Data":"01723246cc1bbae34a1190adcb6127f50693bc58bb85fc7f832013eab635037e"} Sep 07 10:28:22.111385 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:22.111259 2565 scope.go:117] "RemoveContainer" containerID="4d5bd0d11783f43566d801d2d7b944459a064dedb1e5420849803987d4a32aaf" Sep 07 10:28:22.135102 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:22.135077 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb"] Sep 07 10:28:22.142049 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:22.142021 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-2d94a5-openshift-default-7c888df4f8-dwhpb"] Sep 07 10:28:22.543491 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:22.543459 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67574581-cb8a-421b-86a3-7998d17dfe6a" path="/var/lib/kubelet/pods/67574581-cb8a-421b-86a3-7998d17dfe6a/volumes" Sep 07 10:28:27.669206 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:27.669155 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz"] Sep 07 10:28:27.671196 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:27.671177 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05ecbf-62gtz"] Sep 07 10:28:28.544759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:28.544727 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="526e5d0a-09da-4d0a-afcc-2ca165fb2b17" path="/var/lib/kubelet/pods/526e5d0a-09da-4d0a-afcc-2ca165fb2b17/volumes" Sep 07 10:28:30.310837 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.310807 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68"] Sep 07 10:28:30.311299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.311212 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526e5d0a-09da-4d0a-afcc-2ca165fb2b17" containerName="revoke-keys" Sep 07 10:28:30.311299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.311228 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="526e5d0a-09da-4d0a-afcc-2ca165fb2b17" containerName="revoke-keys" Sep 07 10:28:30.311299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.311252 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67574581-cb8a-421b-86a3-7998d17dfe6a" containerName="istio-proxy" Sep 07 10:28:30.311299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.311258 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67574581-cb8a-421b-86a3-7998d17dfe6a" containerName="istio-proxy" Sep 07 10:28:30.311299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.311267 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" Sep 07 10:28:30.311299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.311272 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="012241eb-9b25-4b5f-9d0d-24d8aeba8485" containerName="istio-proxy" Sep 07 10:28:30.311513 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.311345 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="526e5d0a-09da-4d0a-afcc-2ca165fb2b17" containerName="revoke-keys" Sep 07 10:28:30.311513 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.311354 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="67574581-cb8a-421b-86a3-7998d17dfe6a" containerName="istio-proxy" Sep 07 10:28:30.316095 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.316060 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.319369 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.319349 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2c28e354-openshift-default-dockercfg-kjvq7\"" Sep 07 10:28:30.330199 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.330179 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68"] Sep 07 10:28:30.442599 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.442559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45dxs\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-kube-api-access-45dxs\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.442813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.442613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-podinfo\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.442813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.442744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-envoy\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.442813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.442784 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-socket\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.442969 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.442827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-token\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.442969 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.442891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-data\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.442969 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.442945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-credential-socket\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.443087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.442974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-certs\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.443087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.443003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba8e7af-55ce-4182-8255-15ae7360ec5b-istiod-ca-cert\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.543525 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45dxs\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-kube-api-access-45dxs\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.543726 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-podinfo\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.543726 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-envoy\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.543726 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-socket\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.543893 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543749 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-token\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.543893 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-data\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.543893 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-credential-socket\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.544045 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-certs\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.544045 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.543928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba8e7af-55ce-4182-8255-15ae7360ec5b-istiod-ca-cert\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.544597 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.544493 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-socket\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.544597 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.544538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-credential-socket\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.544800 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.544778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-data\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.544859 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.544830 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-certs\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.548870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.547449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-envoy\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.549523 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.549490 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-podinfo\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.552731 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.552703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba8e7af-55ce-4182-8255-15ae7360ec5b-istiod-ca-cert\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.553032 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.553010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45dxs\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-kube-api-access-45dxs\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.554693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.554670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-token\") pod \"e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.627027 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.626927 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:30.928222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.928194 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68"] Sep 07 10:28:30.930510 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:28:30.930477 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba8e7af_55ce_4182_8255_15ae7360ec5b.slice/crio-b7ae826ee78df715674402d93427a51ce5bd6e2e2ff5e9469a2cde986538b609 WatchSource:0}: Error finding container b7ae826ee78df715674402d93427a51ce5bd6e2e2ff5e9469a2cde986538b609: Status 404 returned error can't find the container with id b7ae826ee78df715674402d93427a51ce5bd6e2e2ff5e9469a2cde986538b609 Sep 07 10:28:30.933240 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.933201 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:30.933341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.933296 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:30.933341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:30.933325 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:31.163725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:31.163691 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" event={"ID":"fba8e7af-55ce-4182-8255-15ae7360ec5b","Type":"ContainerStarted","Data":"3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54"} Sep 07 10:28:31.163874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:31.163741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" event={"ID":"fba8e7af-55ce-4182-8255-15ae7360ec5b","Type":"ContainerStarted","Data":"b7ae826ee78df715674402d93427a51ce5bd6e2e2ff5e9469a2cde986538b609"} Sep 07 10:28:31.190729 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:31.190621 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podStartSLOduration=1.190596807 podStartE2EDuration="1.190596807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:31.187766999 +0000 UTC m=+1033.174047335" watchObservedRunningTime="2026-09-07 10:28:31.190596807 +0000 UTC m=+1033.176877142" Sep 07 10:28:31.627382 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:31.627344 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:31.632431 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:31.632410 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:31.801610 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:31.801569 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" podUID="6163f641-f733-4060-809f-5d2e6dfbb8e9" containerName="main" containerID="cri-o://6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2" gracePeriod=45 Sep 07 10:28:32.059005 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.058979 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:28:32.157926 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.157892 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-model-cache\") pod \"6163f641-f733-4060-809f-5d2e6dfbb8e9\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " Sep 07 10:28:32.158103 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.157943 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jlkv\" (UniqueName: \"kubernetes.io/projected/6163f641-f733-4060-809f-5d2e6dfbb8e9-kube-api-access-9jlkv\") pod \"6163f641-f733-4060-809f-5d2e6dfbb8e9\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " Sep 07 10:28:32.158103 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.157968 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-tmp-dir\") pod \"6163f641-f733-4060-809f-5d2e6dfbb8e9\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " Sep 07 10:28:32.158224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158143 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6163f641-f733-4060-809f-5d2e6dfbb8e9-tls-certs\") pod \"6163f641-f733-4060-809f-5d2e6dfbb8e9\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " Sep 07 10:28:32.158273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158226 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6163f641-f733-4060-809f-5d2e6dfbb8e9" (UID: "6163f641-f733-4060-809f-5d2e6dfbb8e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:32.158273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158240 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6163f641-f733-4060-809f-5d2e6dfbb8e9" (UID: "6163f641-f733-4060-809f-5d2e6dfbb8e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:32.158340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158295 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-home\") pod \"6163f641-f733-4060-809f-5d2e6dfbb8e9\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " Sep 07 10:28:32.158340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158325 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-dshm\") pod \"6163f641-f733-4060-809f-5d2e6dfbb8e9\" (UID: \"6163f641-f733-4060-809f-5d2e6dfbb8e9\") " Sep 07 10:28:32.158527 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158504 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-home" (OuterVolumeSpecName: "home") pod "6163f641-f733-4060-809f-5d2e6dfbb8e9" (UID: "6163f641-f733-4060-809f-5d2e6dfbb8e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:32.158742 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158724 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:32.158742 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158742 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:32.158879 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.158750 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:32.160454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.160427 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6163f641-f733-4060-809f-5d2e6dfbb8e9-kube-api-access-9jlkv" (OuterVolumeSpecName: "kube-api-access-9jlkv") pod "6163f641-f733-4060-809f-5d2e6dfbb8e9" (UID: "6163f641-f733-4060-809f-5d2e6dfbb8e9"). InnerVolumeSpecName "kube-api-access-9jlkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:28:32.160537 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.160452 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-dshm" (OuterVolumeSpecName: "dshm") pod "6163f641-f733-4060-809f-5d2e6dfbb8e9" (UID: "6163f641-f733-4060-809f-5d2e6dfbb8e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:32.160537 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.160515 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6163f641-f733-4060-809f-5d2e6dfbb8e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6163f641-f733-4060-809f-5d2e6dfbb8e9" (UID: "6163f641-f733-4060-809f-5d2e6dfbb8e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:28:32.171808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.171739 2565 generic.go:356] "Generic (PLEG): container finished" podID="6163f641-f733-4060-809f-5d2e6dfbb8e9" containerID="6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2" exitCode=0 Sep 07 10:28:32.171808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.171787 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" Sep 07 10:28:32.171947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.171817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" event={"ID":"6163f641-f733-4060-809f-5d2e6dfbb8e9","Type":"ContainerDied","Data":"6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2"} Sep 07 10:28:32.171947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.171864 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn" event={"ID":"6163f641-f733-4060-809f-5d2e6dfbb8e9","Type":"ContainerDied","Data":"9bc8e2fbe278144b1adf9b5bacd512cf47e5b3a581a85c61709de4e7729f087b"} Sep 07 10:28:32.171947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.171884 2565 scope.go:117] "RemoveContainer" containerID="6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2" Sep 07 10:28:32.172385 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.172315 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:32.173497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.173480 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:32.181512 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.181496 2565 scope.go:117] "RemoveContainer" containerID="6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2" Sep 07 10:28:32.181783 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:28:32.181767 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2\": container with ID starting with 6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2 not found: ID does not exist" containerID="6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2" Sep 07 10:28:32.181825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.181792 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2"} err="failed to get container status \"6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2\": rpc error: code = NotFound desc = could not find container \"6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2\": container with ID starting with 6b3c45e529e575ef95e0b67fa5f3d622947222ca18d5ff7c00a0845f473b09c2 not found: ID does not exist" Sep 07 10:28:32.228908 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.228854 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn"] Sep 07 10:28:32.240258 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.240230 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f17c1f/test-model-f17c1f-kserve-b7cf4d48f-77njn"] Sep 07 10:28:32.260266 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.260241 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6163f641-f733-4060-809f-5d2e6dfbb8e9-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:32.260266 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.260263 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jlkv\" (UniqueName: \"kubernetes.io/projected/6163f641-f733-4060-809f-5d2e6dfbb8e9-kube-api-access-9jlkv\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:32.260266 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.260273 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6163f641-f733-4060-809f-5d2e6dfbb8e9-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:32.545239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:32.545204 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6163f641-f733-4060-809f-5d2e6dfbb8e9" path="/var/lib/kubelet/pods/6163f641-f733-4060-809f-5d2e6dfbb8e9/volumes" Sep 07 10:28:34.600281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.600243 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml"] Sep 07 10:28:34.600696 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.600503 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" containerID="cri-o://6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238" gracePeriod=30 Sep 07 10:28:34.928016 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.927914 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m"] Sep 07 10:28:34.928615 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.928581 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6163f641-f733-4060-809f-5d2e6dfbb8e9" containerName="main" Sep 07 10:28:34.928615 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.928608 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6163f641-f733-4060-809f-5d2e6dfbb8e9" containerName="main" Sep 07 10:28:34.928806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.928717 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6163f641-f733-4060-809f-5d2e6dfbb8e9" containerName="main" Sep 07 10:28:34.933898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.933868 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" Sep 07 10:28:34.936589 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.936562 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m"] Sep 07 10:28:34.938564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.938541 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:28:34.938968 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:34.938950 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:28:35.091475 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:35.091443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc43047-f8d6-402e-959c-adf93af6711b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m\" (UID: \"2bc43047-f8d6-402e-959c-adf93af6711b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" Sep 07 10:28:35.192562 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:35.192483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc43047-f8d6-402e-959c-adf93af6711b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m\" (UID: \"2bc43047-f8d6-402e-959c-adf93af6711b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" Sep 07 10:28:35.193065 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:35.193048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc43047-f8d6-402e-959c-adf93af6711b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m\" (UID: \"2bc43047-f8d6-402e-959c-adf93af6711b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" Sep 07 10:28:35.252245 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:35.252218 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" Sep 07 10:28:35.529274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:35.529253 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m"] Sep 07 10:28:35.531027 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:28:35.530993 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc43047_f8d6_402e_959c_adf93af6711b.slice/crio-02f91b50f6858cfff5639c1d75a87d9da456f2c8c875e8c0d3f66f43bb5d0896 WatchSource:0}: Error finding container 02f91b50f6858cfff5639c1d75a87d9da456f2c8c875e8c0d3f66f43bb5d0896: Status 404 returned error can't find the container with id 02f91b50f6858cfff5639c1d75a87d9da456f2c8c875e8c0d3f66f43bb5d0896 Sep 07 10:28:36.199575 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:36.199481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" event={"ID":"2bc43047-f8d6-402e-959c-adf93af6711b","Type":"ContainerStarted","Data":"b4426a6d4f465f6f96cc36b4fc9be5a8ef81f33fbf7ed4321a02c7e27306ca82"} Sep 07 10:28:36.199575 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:36.199543 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" event={"ID":"2bc43047-f8d6-402e-959c-adf93af6711b","Type":"ContainerStarted","Data":"02f91b50f6858cfff5639c1d75a87d9da456f2c8c875e8c0d3f66f43bb5d0896"} Sep 07 10:28:36.216752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:36.216710 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" podStartSLOduration=2.216695921 podStartE2EDuration="2.216695921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:36.215098248 +0000 UTC m=+1038.201378595" watchObservedRunningTime="2026-09-07 10:28:36.216695921 +0000 UTC m=+1038.202976255" Sep 07 10:28:37.206829 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:37.206794 2565 generic.go:356] "Generic (PLEG): container finished" podID="2bc43047-f8d6-402e-959c-adf93af6711b" containerID="b4426a6d4f465f6f96cc36b4fc9be5a8ef81f33fbf7ed4321a02c7e27306ca82" exitCode=0 Sep 07 10:28:37.207201 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:37.206856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" event={"ID":"2bc43047-f8d6-402e-959c-adf93af6711b","Type":"ContainerDied","Data":"b4426a6d4f465f6f96cc36b4fc9be5a8ef81f33fbf7ed4321a02c7e27306ca82"} Sep 07 10:28:38.347616 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:38.347592 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" Sep 07 10:28:38.425313 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:38.425281 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc43047-f8d6-402e-959c-adf93af6711b-maas-api-service-ca\") pod \"2bc43047-f8d6-402e-959c-adf93af6711b\" (UID: \"2bc43047-f8d6-402e-959c-adf93af6711b\") " Sep 07 10:28:38.425691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:38.425671 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc43047-f8d6-402e-959c-adf93af6711b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bc43047-f8d6-402e-959c-adf93af6711b" (UID: "2bc43047-f8d6-402e-959c-adf93af6711b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:28:38.527100 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:38.527068 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc43047-f8d6-402e-959c-adf93af6711b-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:39.220243 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:39.220197 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" event={"ID":"2bc43047-f8d6-402e-959c-adf93af6711b","Type":"ContainerDied","Data":"02f91b50f6858cfff5639c1d75a87d9da456f2c8c875e8c0d3f66f43bb5d0896"} Sep 07 10:28:39.220243 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:39.220219 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m" Sep 07 10:28:39.220521 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:39.220274 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02f91b50f6858cfff5639c1d75a87d9da456f2c8c875e8c0d3f66f43bb5d0896" Sep 07 10:28:39.893098 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:39.893075 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:28:40.058907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.058872 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639c2568-bb85-49cd-9a30-e228190a0acf-istiod-ca-cert\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.058907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.058905 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639c2568-bb85-49cd-9a30-e228190a0acf-istio-podinfo\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.059178 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.058942 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-envoy\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.059178 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059131 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-data\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.059305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059195 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-credential-socket\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.059305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059224 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-istio-token\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.059305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059258 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639c2568-bb85-49cd-9a30-e228190a0acf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:28:40.059305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059272 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m42k8\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-kube-api-access-m42k8\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.059474 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059363 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-data" (OuterVolumeSpecName: "istio-data") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:40.059474 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059422 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-socket\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.059474 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059450 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-certs\") pod \"639c2568-bb85-49cd-9a30-e228190a0acf\" (UID: \"639c2568-bb85-49cd-9a30-e228190a0acf\") " Sep 07 10:28:40.059624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059484 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:40.059845 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059632 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:40.059845 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059785 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:40.059955 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059883 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.059955 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059900 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.059955 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059914 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.059955 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059927 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.059955 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.059939 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639c2568-bb85-49cd-9a30-e228190a0acf-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.061622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.061517 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:40.061622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.061597 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-istio-token" (OuterVolumeSpecName: "istio-token") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:28:40.061880 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.061863 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-kube-api-access-m42k8" (OuterVolumeSpecName: "kube-api-access-m42k8") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "kube-api-access-m42k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:28:40.061982 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.061969 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/639c2568-bb85-49cd-9a30-e228190a0acf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "639c2568-bb85-49cd-9a30-e228190a0acf" (UID: "639c2568-bb85-49cd-9a30-e228190a0acf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:28:40.160973 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.160878 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42k8\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-kube-api-access-m42k8\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.160973 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.160914 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639c2568-bb85-49cd-9a30-e228190a0acf-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.160973 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.160928 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639c2568-bb85-49cd-9a30-e228190a0acf-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.160973 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.160936 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639c2568-bb85-49cd-9a30-e228190a0acf-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:40.227469 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.227434 2565 generic.go:356] "Generic (PLEG): container finished" podID="639c2568-bb85-49cd-9a30-e228190a0acf" containerID="6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238" exitCode=0 Sep 07 10:28:40.227655 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.227478 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" event={"ID":"639c2568-bb85-49cd-9a30-e228190a0acf","Type":"ContainerDied","Data":"6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238"} Sep 07 10:28:40.227655 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.227480 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" Sep 07 10:28:40.227655 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.227529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml" event={"ID":"639c2568-bb85-49cd-9a30-e228190a0acf","Type":"ContainerDied","Data":"704f1bce154bfbf1c3b4ef838c2e88fb0088b64f42dbf60ba97ebd296425177a"} Sep 07 10:28:40.227655 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.227553 2565 scope.go:117] "RemoveContainer" containerID="6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238" Sep 07 10:28:40.242132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.242098 2565 scope.go:117] "RemoveContainer" containerID="9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254" Sep 07 10:28:40.261797 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.261773 2565 scope.go:117] "RemoveContainer" containerID="6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238" Sep 07 10:28:40.262408 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:28:40.262383 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238\": container with ID starting with 6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238 not found: ID does not exist" containerID="6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238" Sep 07 10:28:40.262572 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.262544 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238"} err="failed to get container status \"6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238\": rpc error: code = NotFound desc = could not find container \"6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238\": container with ID starting with 6ae80c32b940b24d0d9dede4a4e829bfd2ce32f7f26c9f059d730dde1f313238 not found: ID does not exist" Sep 07 10:28:40.262679 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.262664 2565 scope.go:117] "RemoveContainer" containerID="9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254" Sep 07 10:28:40.263290 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:28:40.263153 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254\": container with ID starting with 9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254 not found: ID does not exist" containerID="9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254" Sep 07 10:28:40.263290 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.263212 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254"} err="failed to get container status \"9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254\": rpc error: code = NotFound desc = could not find container \"9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254\": container with ID starting with 9080f932b83d43fd3a5d2b79c6e89b301f324418872faec8ad592e7fb290b254 not found: ID does not exist" Sep 07 10:28:40.265024 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.264967 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml"] Sep 07 10:28:40.267841 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.267820 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-05ecbf-openshift-default-7dd9fffcd8-ccrml"] Sep 07 10:28:40.549263 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:40.549225 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" path="/var/lib/kubelet/pods/639c2568-bb85-49cd-9a30-e228190a0acf/volumes" Sep 07 10:28:45.814474 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:45.814441 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m"] Sep 07 10:28:45.818914 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:45.818890 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f17c1f-4jc6m"] Sep 07 10:28:46.544980 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:46.544939 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc43047-f8d6-402e-959c-adf93af6711b" path="/var/lib/kubelet/pods/2bc43047-f8d6-402e-959c-adf93af6711b/volumes" Sep 07 10:28:47.529506 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:47.527976 2565 patch_prober.go:28] interesting pod/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:42758->10.132.0.59:15021: read: connection reset by peer" start-of-body= Sep 07 10:28:47.529506 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:47.528063 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:42758->10.132.0.59:15021: read: connection reset by peer" Sep 07 10:28:48.286446 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:48.286408 2565 generic.go:356] "Generic (PLEG): container finished" podID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerID="3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54" exitCode=0 Sep 07 10:28:48.286627 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:48.286480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" event={"ID":"fba8e7af-55ce-4182-8255-15ae7360ec5b","Type":"ContainerDied","Data":"3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54"} Sep 07 10:28:48.286892 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:48.286879 2565 scope.go:117] "RemoveContainer" containerID="3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54" Sep 07 10:28:48.287884 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:48.287858 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:48.287938 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:48.287923 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:48.287976 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:48.287953 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:49.293609 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:49.293571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" event={"ID":"fba8e7af-55ce-4182-8255-15ae7360ec5b","Type":"ContainerStarted","Data":"5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf"} Sep 07 10:28:49.628125 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:49.628040 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:49.628314 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:49.628270 2565 patch_prober.go:28] interesting pod/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 10:28:49.628360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:49.628321 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 10:28:50.627946 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:50.627905 2565 patch_prober.go:28] interesting pod/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 10:28:50.628381 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:50.627961 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 10:28:51.628178 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:51.628124 2565 patch_prober.go:28] interesting pod/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 10:28:51.628590 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:51.628234 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 10:28:52.606765 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:52.606737 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht"] Sep 07 10:28:52.607014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:52.606951 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" containerID="cri-o://fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4" gracePeriod=30 Sep 07 10:28:52.627557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:52.627465 2565 patch_prober.go:28] interesting pod/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 10:28:52.627557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:52.627545 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 10:28:53.456183 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.455987 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7"] Sep 07 10:28:53.456652 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.456634 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" Sep 07 10:28:53.456721 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.456654 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" Sep 07 10:28:53.456721 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.456664 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc43047-f8d6-402e-959c-adf93af6711b" containerName="revoke-keys" Sep 07 10:28:53.456721 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.456672 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc43047-f8d6-402e-959c-adf93af6711b" containerName="revoke-keys" Sep 07 10:28:53.456880 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.456790 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc43047-f8d6-402e-959c-adf93af6711b" containerName="revoke-keys" Sep 07 10:28:53.456880 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.456804 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" Sep 07 10:28:53.456880 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.456814 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" Sep 07 10:28:53.461352 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.460663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.465298 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.465258 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0a577c-openshift-default-dockercfg-sxzkz\"" Sep 07 10:28:53.471822 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.471791 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7"] Sep 07 10:28:53.505608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-credential-socket\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.505805 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-data\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.505805 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-podinfo\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.505805 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505701 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-token\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.505805 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-socket\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.506012 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-certs\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.506012 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b52e926-8672-4aa0-9402-2492fd59dc88-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.506012 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-envoy\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.506012 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.505940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnvn2\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-kube-api-access-gnvn2\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607080 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607045 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-socket\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-certs\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b52e926-8672-4aa0-9402-2492fd59dc88-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-envoy\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607310 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvn2\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-kube-api-access-gnvn2\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607540 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-credential-socket\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607540 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-data\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607540 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607493 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-certs\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607705 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-podinfo\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607705 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-token\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607817 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607712 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-data\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607817 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607752 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-credential-socket\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607940 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607866 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-socket\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.607940 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.607886 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b52e926-8672-4aa0-9402-2492fd59dc88-istiod-ca-cert\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.610025 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.610001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-envoy\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.610195 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.610179 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-podinfo\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.615692 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.615665 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-token\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.615803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.615780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvn2\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-kube-api-access-gnvn2\") pod \"e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:53.627741 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.627715 2565 patch_prober.go:28] interesting pod/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 07 10:28:53.627843 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.627770 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 07 10:28:53.781600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:53.781566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:54.084207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.084184 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7"] Sep 07 10:28:54.086219 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:28:54.086194 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b52e926_8672_4aa0_9402_2492fd59dc88.slice/crio-10c9e048726174d0b941f8fe580c971ad369e51b34403b30f04991a7615332a8 WatchSource:0}: Error finding container 10c9e048726174d0b941f8fe580c971ad369e51b34403b30f04991a7615332a8: Status 404 returned error can't find the container with id 10c9e048726174d0b941f8fe580c971ad369e51b34403b30f04991a7615332a8 Sep 07 10:28:54.088827 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.088796 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:54.088910 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.088857 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:54.088910 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.088889 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:28:54.323657 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.323625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" event={"ID":"0b52e926-8672-4aa0-9402-2492fd59dc88","Type":"ContainerStarted","Data":"75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1"} Sep 07 10:28:54.323855 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.323675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" event={"ID":"0b52e926-8672-4aa0-9402-2492fd59dc88","Type":"ContainerStarted","Data":"10c9e048726174d0b941f8fe580c971ad369e51b34403b30f04991a7615332a8"} Sep 07 10:28:54.346639 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.346509 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" podStartSLOduration=1.346486434 podStartE2EDuration="1.346486434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:54.342430347 +0000 UTC m=+1056.328710682" watchObservedRunningTime="2026-09-07 10:28:54.346486434 +0000 UTC m=+1056.332766768" Sep 07 10:28:54.630875 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.630799 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:54.631292 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.631277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:54.632049 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.632028 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:28:54.782047 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:54.782012 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:55.788104 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:55.788070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:56.335515 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:56.335474 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:56.336407 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:56.336382 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:28:57.869752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.869690 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:28:57.951835 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.951806 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-credential-socket\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.951848 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-podinfo\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.951871 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-socket\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.951981 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-data\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952100 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-certs\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952102 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:57.952214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952134 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-envoy\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952196 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:57.952423 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952203 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-token\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952423 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952247 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istiod-ca-cert\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952423 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952293 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:57.952423 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952346 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c59fb\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-kube-api-access-c59fb\") pod \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\" (UID: \"a81003c8-f0db-4886-9c81-ab16bf4a4b02\") " Sep 07 10:28:57.952678 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952656 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-data" (OuterVolumeSpecName: "istio-data") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:57.952818 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952713 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:28:57.952818 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952740 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:57.952818 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952757 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:57.952818 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.952789 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:57.954407 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.954383 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-token" (OuterVolumeSpecName: "istio-token") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:28:57.954539 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.954520 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:28:57.954588 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.954552 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-kube-api-access-c59fb" (OuterVolumeSpecName: "kube-api-access-c59fb") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "kube-api-access-c59fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:28:57.954760 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:57.954745 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a81003c8-f0db-4886-9c81-ab16bf4a4b02" (UID: "a81003c8-f0db-4886-9c81-ab16bf4a4b02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:28:58.054080 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.054046 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c59fb\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-kube-api-access-c59fb\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:58.054080 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.054077 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:58.054080 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.054087 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:58.054326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.054095 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:58.054326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.054104 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:58.054326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.054112 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81003c8-f0db-4886-9c81-ab16bf4a4b02-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:28:58.351835 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.351802 2565 generic.go:356] "Generic (PLEG): container finished" podID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerID="fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4" exitCode=0 Sep 07 10:28:58.352041 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.351860 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" Sep 07 10:28:58.352041 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.351882 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" event={"ID":"a81003c8-f0db-4886-9c81-ab16bf4a4b02","Type":"ContainerDied","Data":"fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4"} Sep 07 10:28:58.352041 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.351944 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht" event={"ID":"a81003c8-f0db-4886-9c81-ab16bf4a4b02","Type":"ContainerDied","Data":"fd45dd17130b742a539cba70cb5e40aa74e1df237920075f699abb1017e96267"} Sep 07 10:28:58.352041 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.351973 2565 scope.go:117] "RemoveContainer" containerID="fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4" Sep 07 10:28:58.364668 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.364648 2565 scope.go:117] "RemoveContainer" containerID="eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb" Sep 07 10:28:58.374480 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.374460 2565 scope.go:117] "RemoveContainer" containerID="fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4" Sep 07 10:28:58.374818 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:28:58.374791 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4\": container with ID starting with fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4 not found: ID does not exist" containerID="fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4" Sep 07 10:28:58.374924 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.374831 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4"} err="failed to get container status \"fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4\": rpc error: code = NotFound desc = could not find container \"fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4\": container with ID starting with fa55b90cab9e3a93fcbefe8a50157ec0f592f0075cd9ac2d4cef3a6bfcd4fad4 not found: ID does not exist" Sep 07 10:28:58.374924 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.374912 2565 scope.go:117] "RemoveContainer" containerID="eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb" Sep 07 10:28:58.375521 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:28:58.375356 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb\": container with ID starting with eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb not found: ID does not exist" containerID="eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb" Sep 07 10:28:58.375521 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.375389 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb"} err="failed to get container status \"eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb\": rpc error: code = NotFound desc = could not find container \"eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb\": container with ID starting with eab988a50ac7f9c17039693a2d0de320dfe0d9d64ef7f19bee5e4f1cdf3516fb not found: ID does not exist" Sep 07 10:28:58.377471 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.377452 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht"] Sep 07 10:28:58.383869 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.383850 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f17c1f-openshift-default-75d98f5684-8jxht"] Sep 07 10:28:58.544170 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:28:58.544122 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" path="/var/lib/kubelet/pods/a81003c8-f0db-4886-9c81-ab16bf4a4b02/volumes" Sep 07 10:29:05.012894 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.012863 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9"] Sep 07 10:29:05.013429 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.013412 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" Sep 07 10:29:05.013493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.013432 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="639c2568-bb85-49cd-9a30-e228190a0acf" containerName="istio-proxy" Sep 07 10:29:05.013493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.013454 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" Sep 07 10:29:05.013493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.013460 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" Sep 07 10:29:05.013493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.013477 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" Sep 07 10:29:05.013493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.013482 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" Sep 07 10:29:05.013761 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.013544 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" Sep 07 10:29:05.013761 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.013555 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81003c8-f0db-4886-9c81-ab16bf4a4b02" containerName="istio-proxy" Sep 07 10:29:05.018098 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.018082 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" Sep 07 10:29:05.020785 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.020727 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:29:05.021711 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.021679 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:29:05.025377 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.025284 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9"] Sep 07 10:29:05.120788 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.120750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b08015-a81e-493c-a4ad-349f40472911-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9\" (UID: \"93b08015-a81e-493c-a4ad-349f40472911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" Sep 07 10:29:05.221878 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.221839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b08015-a81e-493c-a4ad-349f40472911-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9\" (UID: \"93b08015-a81e-493c-a4ad-349f40472911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" Sep 07 10:29:05.222482 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.222463 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b08015-a81e-493c-a4ad-349f40472911-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9\" (UID: \"93b08015-a81e-493c-a4ad-349f40472911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" Sep 07 10:29:05.331017 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.330920 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" Sep 07 10:29:05.621168 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:05.621130 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9"] Sep 07 10:29:05.622743 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:29:05.622717 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b08015_a81e_493c_a4ad_349f40472911.slice/crio-307cb172fdfd4a4895e58a942822304cca92ec822ddbb464c2f2a773263f9866 WatchSource:0}: Error finding container 307cb172fdfd4a4895e58a942822304cca92ec822ddbb464c2f2a773263f9866: Status 404 returned error can't find the container with id 307cb172fdfd4a4895e58a942822304cca92ec822ddbb464c2f2a773263f9866 Sep 07 10:29:06.399258 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:06.399227 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" event={"ID":"93b08015-a81e-493c-a4ad-349f40472911","Type":"ContainerStarted","Data":"f591a88f734e3c51e4e29bd408e5f3464c639c5b2daa23c06df453a48dca2062"} Sep 07 10:29:06.399663 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:06.399278 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" event={"ID":"93b08015-a81e-493c-a4ad-349f40472911","Type":"ContainerStarted","Data":"307cb172fdfd4a4895e58a942822304cca92ec822ddbb464c2f2a773263f9866"} Sep 07 10:29:06.418731 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:06.418678 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" podStartSLOduration=2.418663556 podStartE2EDuration="2.418663556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:06.417848782 +0000 UTC m=+1068.404129117" watchObservedRunningTime="2026-09-07 10:29:06.418663556 +0000 UTC m=+1068.404943886" Sep 07 10:29:07.407009 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:07.406974 2565 generic.go:356] "Generic (PLEG): container finished" podID="93b08015-a81e-493c-a4ad-349f40472911" containerID="f591a88f734e3c51e4e29bd408e5f3464c639c5b2daa23c06df453a48dca2062" exitCode=0 Sep 07 10:29:07.407375 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:07.407060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" event={"ID":"93b08015-a81e-493c-a4ad-349f40472911","Type":"ContainerDied","Data":"f591a88f734e3c51e4e29bd408e5f3464c639c5b2daa23c06df453a48dca2062"} Sep 07 10:29:08.559498 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:08.559472 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" Sep 07 10:29:08.658079 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:08.658045 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b08015-a81e-493c-a4ad-349f40472911-maas-api-service-ca\") pod \"93b08015-a81e-493c-a4ad-349f40472911\" (UID: \"93b08015-a81e-493c-a4ad-349f40472911\") " Sep 07 10:29:08.658441 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:08.658418 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b08015-a81e-493c-a4ad-349f40472911-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93b08015-a81e-493c-a4ad-349f40472911" (UID: "93b08015-a81e-493c-a4ad-349f40472911"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:29:08.759578 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:08.759547 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b08015-a81e-493c-a4ad-349f40472911-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:09.420124 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:09.420041 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" Sep 07 10:29:09.420124 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:09.420092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9" event={"ID":"93b08015-a81e-493c-a4ad-349f40472911","Type":"ContainerDied","Data":"307cb172fdfd4a4895e58a942822304cca92ec822ddbb464c2f2a773263f9866"} Sep 07 10:29:09.420340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:09.420151 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307cb172fdfd4a4895e58a942822304cca92ec822ddbb464c2f2a773263f9866" Sep 07 10:29:10.426977 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:10.426893 2565 generic.go:356] "Generic (PLEG): container finished" podID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerID="75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1" exitCode=0 Sep 07 10:29:10.427504 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:10.426973 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" event={"ID":"0b52e926-8672-4aa0-9402-2492fd59dc88","Type":"ContainerDied","Data":"75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1"} Sep 07 10:29:10.427616 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:10.427598 2565 scope.go:117] "RemoveContainer" containerID="75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1" Sep 07 10:29:10.428707 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:10.428677 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:10.428781 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:10.428765 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:10.428822 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:10.428805 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:11.339197 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:11.335955 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:29:11.435882 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:11.435839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" event={"ID":"0b52e926-8672-4aa0-9402-2492fd59dc88","Type":"ContainerStarted","Data":"98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b"} Sep 07 10:29:11.782127 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:11.782083 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:29:11.783483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:11.783453 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:29:11.783633 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:11.783520 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 07 10:29:12.782317 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:12.782275 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:29:12.782798 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:12.782337 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 07 10:29:13.783047 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:13.783003 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:29:13.783455 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:13.783064 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 07 10:29:14.782573 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:14.782536 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:29:14.782763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:14.782615 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 07 10:29:15.664595 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:15.664555 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9"] Sep 07 10:29:15.666836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:15.666808 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2c28e354-v2gz9"] Sep 07 10:29:15.782565 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:15.782525 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:29:15.782764 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:15.782616 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 07 10:29:16.543808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:16.543772 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b08015-a81e-493c-a4ad-349f40472911" path="/var/lib/kubelet/pods/93b08015-a81e-493c-a4ad-349f40472911/volumes" Sep 07 10:29:16.787085 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:16.787058 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:29:16.787473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:16.787439 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:29:16.788212 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:16.788193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:29:21.741999 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.741968 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp"] Sep 07 10:29:21.742457 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.742442 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b08015-a81e-493c-a4ad-349f40472911" containerName="revoke-keys" Sep 07 10:29:21.742498 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.742459 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b08015-a81e-493c-a4ad-349f40472911" containerName="revoke-keys" Sep 07 10:29:21.742538 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.742528 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b08015-a81e-493c-a4ad-349f40472911" containerName="revoke-keys" Sep 07 10:29:21.745511 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.745492 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.748693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.748662 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-93e41e-openshift-default-dockercfg-54qjc\"" Sep 07 10:29:21.761752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.761732 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp"] Sep 07 10:29:21.806059 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806029 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918e9d-81f9-4f8f-9905-f4e91f738770-istiod-ca-cert\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.806263 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-data\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.806263 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806142 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-credential-socket\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.806263 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-envoy\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.806263 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-socket\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.806550 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806293 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-certs\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.806550 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-token\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.806550 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-podinfo\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.806550 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.806410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqvz\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-kube-api-access-8zqvz\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907471 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-credential-socket\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907486 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-envoy\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907520 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-socket\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-certs\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-token\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-podinfo\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907831 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-credential-socket\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907833 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-certs\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqvz\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-kube-api-access-8zqvz\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907895 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-socket\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.907990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.907958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918e9d-81f9-4f8f-9905-f4e91f738770-istiod-ca-cert\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.908246 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.908019 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-data\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.908409 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.908335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-data\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.908599 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.908571 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918e9d-81f9-4f8f-9905-f4e91f738770-istiod-ca-cert\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.910021 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.909996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-envoy\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.910284 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.910269 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-podinfo\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.916070 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.916048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-token\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:21.916652 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:21.916626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqvz\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-kube-api-access-8zqvz\") pod \"e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:22.055867 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.055777 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:22.343805 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.343780 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp"] Sep 07 10:29:22.344316 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:29:22.344291 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a918e9d_81f9_4f8f_9905_f4e91f738770.slice/crio-ea8a5e6b246f8167ee00d651efb86a5f5090bebb79f0269a64f09dbfafc166c0 WatchSource:0}: Error finding container ea8a5e6b246f8167ee00d651efb86a5f5090bebb79f0269a64f09dbfafc166c0: Status 404 returned error can't find the container with id ea8a5e6b246f8167ee00d651efb86a5f5090bebb79f0269a64f09dbfafc166c0 Sep 07 10:29:22.346586 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.346551 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:22.346671 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.346612 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:22.346671 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.346640 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:22.531891 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.531846 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" event={"ID":"1a918e9d-81f9-4f8f-9905-f4e91f738770","Type":"ContainerStarted","Data":"978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa"} Sep 07 10:29:22.532107 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.531915 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" event={"ID":"1a918e9d-81f9-4f8f-9905-f4e91f738770","Type":"ContainerStarted","Data":"ea8a5e6b246f8167ee00d651efb86a5f5090bebb79f0269a64f09dbfafc166c0"} Sep 07 10:29:22.536855 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.536826 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68"] Sep 07 10:29:22.537099 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.537076 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" containerID="cri-o://5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf" gracePeriod=30 Sep 07 10:29:22.572663 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:22.572584 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" podStartSLOduration=1.572563315 podStartE2EDuration="1.572563315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:22.571279308 +0000 UTC m=+1084.557559642" watchObservedRunningTime="2026-09-07 10:29:22.572563315 +0000 UTC m=+1084.558843650" Sep 07 10:29:23.056293 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:23.056263 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:23.061454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:23.061431 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:23.537915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:23.537879 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:23.538867 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:23.538849 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:24.633662 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:24.633625 2565 patch_prober.go:28] interesting pod/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 10:29:24.634037 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:24.633696 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:29:27.791090 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.791065 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:29:27.864931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.864851 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-credential-socket\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865065 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.864980 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-certs\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865065 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865019 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45dxs\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-kube-api-access-45dxs\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865065 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865047 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-token\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865199 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865088 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba8e7af-55ce-4182-8255-15ae7360ec5b-istiod-ca-cert\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865199 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865138 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-data\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865199 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865136 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:29:27.865359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865244 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-envoy\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865265 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:29:27.865359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865287 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-podinfo\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865313 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-socket\") pod \"fba8e7af-55ce-4182-8255-15ae7360ec5b\" (UID: \"fba8e7af-55ce-4182-8255-15ae7360ec5b\") " Sep 07 10:29:27.865573 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865425 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:29:27.865659 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865624 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:29:27.865741 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865722 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:27.865811 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865746 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:27.865811 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865760 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:27.865811 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865773 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:27.865957 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.865840 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba8e7af-55ce-4182-8255-15ae7360ec5b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:29:27.867419 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.867397 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:29:27.867890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.867871 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:29:27.867972 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.867898 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-kube-api-access-45dxs" (OuterVolumeSpecName: "kube-api-access-45dxs") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "kube-api-access-45dxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:29:27.868032 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.868002 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fba8e7af-55ce-4182-8255-15ae7360ec5b" (UID: "fba8e7af-55ce-4182-8255-15ae7360ec5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:29:27.966808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.966768 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45dxs\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-kube-api-access-45dxs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:27.966808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.966802 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:27.966808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.966811 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba8e7af-55ce-4182-8255-15ae7360ec5b-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:27.966808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.966818 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:27.967092 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:27.966826 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba8e7af-55ce-4182-8255-15ae7360ec5b-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:29:28.568146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.568117 2565 generic.go:356] "Generic (PLEG): container finished" podID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerID="5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf" exitCode=0 Sep 07 10:29:28.568291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.568180 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" event={"ID":"fba8e7af-55ce-4182-8255-15ae7360ec5b","Type":"ContainerDied","Data":"5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf"} Sep 07 10:29:28.568291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.568217 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" event={"ID":"fba8e7af-55ce-4182-8255-15ae7360ec5b","Type":"ContainerDied","Data":"b7ae826ee78df715674402d93427a51ce5bd6e2e2ff5e9469a2cde986538b609"} Sep 07 10:29:28.568291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.568242 2565 scope.go:117] "RemoveContainer" containerID="5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf" Sep 07 10:29:28.568291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.568184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68" Sep 07 10:29:28.577252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.577238 2565 scope.go:117] "RemoveContainer" containerID="3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54" Sep 07 10:29:28.586005 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.585989 2565 scope.go:117] "RemoveContainer" containerID="5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf" Sep 07 10:29:28.586278 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:29:28.586260 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf\": container with ID starting with 5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf not found: ID does not exist" containerID="5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf" Sep 07 10:29:28.586355 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.586287 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf"} err="failed to get container status \"5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf\": rpc error: code = NotFound desc = could not find container \"5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf\": container with ID starting with 5bd1d514a2f09a4806d0a9e3a1a3de51fc68d32f4e7dcff5843b282fd42d6daf not found: ID does not exist" Sep 07 10:29:28.586355 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.586304 2565 scope.go:117] "RemoveContainer" containerID="3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54" Sep 07 10:29:28.586556 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:29:28.586540 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54\": container with ID starting with 3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54 not found: ID does not exist" containerID="3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54" Sep 07 10:29:28.586611 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.586561 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54"} err="failed to get container status \"3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54\": rpc error: code = NotFound desc = could not find container \"3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54\": container with ID starting with 3b5ff9b205b3df959fada2c2f708cf5460625cb3e27bfa4bb35baa749c196c54 not found: ID does not exist" Sep 07 10:29:28.586808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.586789 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68"] Sep 07 10:29:28.593180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:28.592655 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2c28e354-openshift-default-776d4fb6d8-cql68"] Sep 07 10:29:30.544593 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:30.544557 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" path="/var/lib/kubelet/pods/fba8e7af-55ce-4182-8255-15ae7360ec5b/volumes" Sep 07 10:29:36.068430 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.068394 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql"] Sep 07 10:29:36.068990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.068973 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" Sep 07 10:29:36.069052 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.068993 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" Sep 07 10:29:36.069052 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.069015 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" Sep 07 10:29:36.069052 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.069021 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" Sep 07 10:29:36.069153 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.069077 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" Sep 07 10:29:36.073672 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.073650 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.076353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.076334 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7bdcc7-openshift-default-dockercfg-zzwpv\"" Sep 07 10:29:36.085547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.085522 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql"] Sep 07 10:29:36.151444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.151407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-socket\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.151741 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.151723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-podinfo\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.151894 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.151878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b9ddb-1092-40f9-a889-5e4107b4d84e-istiod-ca-cert\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.152054 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.152039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsj5d\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-kube-api-access-hsj5d\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.152225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.152205 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-envoy\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.152380 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.152364 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-token\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.152557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.152542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-data\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.152704 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.152689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-credential-socket\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.152844 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.152829 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-certs\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.253802 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.253759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-data\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.253822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-credential-socket\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.253854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-certs\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.253933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-socket\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.253976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-podinfo\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.254006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b9ddb-1092-40f9-a889-5e4107b4d84e-istiod-ca-cert\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.254059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsj5d\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-kube-api-access-hsj5d\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.254100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-envoy\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.254122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-token\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.254931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.254901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-data\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.255207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.255182 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-credential-socket\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.255446 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.255424 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-certs\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.255671 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.255649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-socket\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.256925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.256896 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b9ddb-1092-40f9-a889-5e4107b4d84e-istiod-ca-cert\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.258674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.258651 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-podinfo\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.259340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.259320 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-envoy\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.262402 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.262386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-token\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.263569 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.263553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsj5d\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-kube-api-access-hsj5d\") pod \"e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.388635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.388541 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:36.679794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.679688 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql"] Sep 07 10:29:36.682824 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:29:36.682800 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661b9ddb_1092_40f9_a889_5e4107b4d84e.slice/crio-2138e918c5189d79b1de3cf1fc4a83a6e3c35ea68fc278979f99704b39bafa35 WatchSource:0}: Error finding container 2138e918c5189d79b1de3cf1fc4a83a6e3c35ea68fc278979f99704b39bafa35: Status 404 returned error can't find the container with id 2138e918c5189d79b1de3cf1fc4a83a6e3c35ea68fc278979f99704b39bafa35 Sep 07 10:29:36.685564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.685532 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:36.685666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.685596 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:36.685666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:36.685626 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:37.628726 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:37.628682 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" event={"ID":"661b9ddb-1092-40f9-a889-5e4107b4d84e","Type":"ContainerStarted","Data":"94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc"} Sep 07 10:29:37.629195 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:37.628747 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" event={"ID":"661b9ddb-1092-40f9-a889-5e4107b4d84e","Type":"ContainerStarted","Data":"2138e918c5189d79b1de3cf1fc4a83a6e3c35ea68fc278979f99704b39bafa35"} Sep 07 10:29:37.649131 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:37.649079 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" podStartSLOduration=1.6490627820000001 podStartE2EDuration="1.649062782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:37.647228014 +0000 UTC m=+1099.633508351" watchObservedRunningTime="2026-09-07 10:29:37.649062782 +0000 UTC m=+1099.635343183" Sep 07 10:29:38.389242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:38.389204 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:38.394719 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:38.394690 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:38.635087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:38.635048 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:38.636091 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:38.636072 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:39.579323 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:39.579281 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:29:39.579508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:39.579339 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:29:40.648315 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:40.648281 2565 generic.go:356] "Generic (PLEG): container finished" podID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerID="978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa" exitCode=0 Sep 07 10:29:40.648848 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:40.648365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" event={"ID":"1a918e9d-81f9-4f8f-9905-f4e91f738770","Type":"ContainerDied","Data":"978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa"} Sep 07 10:29:40.648848 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:40.648825 2565 scope.go:117] "RemoveContainer" containerID="978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa" Sep 07 10:29:40.650244 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:40.650207 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:40.650331 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:40.650302 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:40.650372 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:40.650348 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:41.663018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:41.662969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" event={"ID":"1a918e9d-81f9-4f8f-9905-f4e91f738770","Type":"ContainerStarted","Data":"37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74"} Sep 07 10:29:42.056735 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:42.056689 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:42.056965 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:42.056836 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:29:42.056965 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:42.056885 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 07 10:29:43.056506 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:43.056456 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:29:43.057028 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:43.056530 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 07 10:29:44.056535 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:44.056489 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:29:44.057052 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:44.056561 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 07 10:29:45.057274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:45.057223 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:29:45.057809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:45.057300 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 07 10:29:46.057216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:46.056867 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:29:46.057216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:46.056938 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 07 10:29:47.077726 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:47.077694 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:47.078235 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:47.078200 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:47.079173 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:47.079142 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:29:54.677354 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:54.677306 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:29:54.677948 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:54.677406 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:29:55.746142 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:55.746109 2565 generic.go:356] "Generic (PLEG): container finished" podID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerID="94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc" exitCode=0 Sep 07 10:29:55.746572 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:55.746185 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" event={"ID":"661b9ddb-1092-40f9-a889-5e4107b4d84e","Type":"ContainerDied","Data":"94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc"} Sep 07 10:29:55.746615 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:55.746586 2565 scope.go:117] "RemoveContainer" containerID="94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc" Sep 07 10:29:55.747637 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:55.747612 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:55.747719 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:55.747678 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:55.747719 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:55.747709 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:29:56.760036 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:56.759999 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" event={"ID":"661b9ddb-1092-40f9-a889-5e4107b4d84e","Type":"ContainerStarted","Data":"44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3"} Sep 07 10:29:57.388981 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:57.388939 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:29:57.389240 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:57.389211 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 07 10:29:57.389323 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:57.389277 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 07 10:29:58.389537 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:58.389501 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 07 10:29:58.389954 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:58.389558 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 07 10:29:59.389002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:59.388962 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 07 10:29:59.389225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:29:59.389022 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 07 10:30:00.138500 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.138456 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62"] Sep 07 10:30:00.139278 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.139241 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba8e7af-55ce-4182-8255-15ae7360ec5b" containerName="istio-proxy" Sep 07 10:30:00.147776 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.147743 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" Sep 07 10:30:00.150238 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.150207 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:30:00.155626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.147639 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq"] Sep 07 10:30:00.159544 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.159513 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z"] Sep 07 10:30:00.161123 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.161097 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" Sep 07 10:30:00.163407 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.163382 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62"] Sep 07 10:30:00.163509 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.163429 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq"] Sep 07 10:30:00.163630 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.163512 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" Sep 07 10:30:00.172257 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.172237 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" Sep 07 10:30:00.181032 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.181015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" Sep 07 10:30:00.192185 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.190311 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z"] Sep 07 10:30:00.230672 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.230644 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" Sep 07 10:30:00.390088 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.389830 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 07 10:30:00.390088 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.389885 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 07 10:30:00.514862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.514827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq"] Sep 07 10:30:00.517370 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.517346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62"] Sep 07 10:30:00.518034 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:30:00.518013 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6453be_7d00_45d5_bac2_37aa0c654c37.slice/crio-9d92e9aee094bdebddc48ee24e4c90b6e65069258d4621ae1a717bd541b3f4d5 WatchSource:0}: Error finding container 9d92e9aee094bdebddc48ee24e4c90b6e65069258d4621ae1a717bd541b3f4d5: Status 404 returned error can't find the container with id 9d92e9aee094bdebddc48ee24e4c90b6e65069258d4621ae1a717bd541b3f4d5 Sep 07 10:30:00.518618 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:30:00.518589 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080e0ecb_3646_4836_b815_6ef146b19137.slice/crio-2c1b22cdc07784a3abf6a67a6f2f042661dc469c3d59ada7674e342230a28a3e WatchSource:0}: Error finding container 2c1b22cdc07784a3abf6a67a6f2f042661dc469c3d59ada7674e342230a28a3e: Status 404 returned error can't find the container with id 2c1b22cdc07784a3abf6a67a6f2f042661dc469c3d59ada7674e342230a28a3e Sep 07 10:30:00.585242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.584624 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z"] Sep 07 10:30:00.794021 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.793989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" event={"ID":"2e6453be-7d00-45d5-bac2-37aa0c654c37","Type":"ContainerStarted","Data":"9d92e9aee094bdebddc48ee24e4c90b6e65069258d4621ae1a717bd541b3f4d5"} Sep 07 10:30:00.795349 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.795324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" event={"ID":"080e0ecb-3646-4836-b815-6ef146b19137","Type":"ContainerStarted","Data":"2c1b22cdc07784a3abf6a67a6f2f042661dc469c3d59ada7674e342230a28a3e"} Sep 07 10:30:00.796581 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:00.796554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" event={"ID":"bdb34e07-90de-4fdb-ab28-a6902bf866d1","Type":"ContainerStarted","Data":"6ece45781806638e34b5242d603355436f621790f9f84c86d970adc7f46e86c3"} Sep 07 10:30:01.390201 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:01.390143 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 07 10:30:01.390691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:01.390226 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 07 10:30:01.805309 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:01.805270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" event={"ID":"2e6453be-7d00-45d5-bac2-37aa0c654c37","Type":"ContainerStarted","Data":"889553c09e503167ea0407fae42f23f90d84060317c405128f2c57b3f0bcd0f0"} Sep 07 10:30:01.807146 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:01.807117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" event={"ID":"080e0ecb-3646-4836-b815-6ef146b19137","Type":"ContainerStarted","Data":"debfff4f6df9d1ec48f18fd53cb1e6db3249e2872158bbf0fdafe9e2c2141174"} Sep 07 10:30:01.808708 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:01.808688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" event={"ID":"bdb34e07-90de-4fdb-ab28-a6902bf866d1","Type":"ContainerStarted","Data":"719aada5c9631a5e00291e9eb8554c439b5dfabd0e6c4a559348f4a7077b815a"} Sep 07 10:30:01.825834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:01.825787 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" podStartSLOduration=1.8257747869999998 podStartE2EDuration="1.825774787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:01.823142205 +0000 UTC m=+1123.809422540" watchObservedRunningTime="2026-09-07 10:30:01.825774787 +0000 UTC m=+1123.812055121" Sep 07 10:30:01.845736 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:01.845680 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" podStartSLOduration=1.8456611600000001 podStartE2EDuration="1.84566116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:01.84455771 +0000 UTC m=+1123.830838104" watchObservedRunningTime="2026-09-07 10:30:01.84566116 +0000 UTC m=+1123.831941498" Sep 07 10:30:01.860823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:01.860655 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" podStartSLOduration=1.8606356339999999 podStartE2EDuration="1.860635634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:01.86029514 +0000 UTC m=+1123.846575476" watchObservedRunningTime="2026-09-07 10:30:01.860635634 +0000 UTC m=+1123.846915970" Sep 07 10:30:02.393371 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.393342 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:30:02.393761 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.393710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:30:02.394496 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.394475 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:30:02.815396 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.815368 2565 generic.go:356] "Generic (PLEG): container finished" podID="2e6453be-7d00-45d5-bac2-37aa0c654c37" containerID="889553c09e503167ea0407fae42f23f90d84060317c405128f2c57b3f0bcd0f0" exitCode=0 Sep 07 10:30:02.815604 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.815465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" event={"ID":"2e6453be-7d00-45d5-bac2-37aa0c654c37","Type":"ContainerDied","Data":"889553c09e503167ea0407fae42f23f90d84060317c405128f2c57b3f0bcd0f0"} Sep 07 10:30:02.817023 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.816998 2565 generic.go:356] "Generic (PLEG): container finished" podID="080e0ecb-3646-4836-b815-6ef146b19137" containerID="debfff4f6df9d1ec48f18fd53cb1e6db3249e2872158bbf0fdafe9e2c2141174" exitCode=0 Sep 07 10:30:02.817139 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.817022 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" event={"ID":"080e0ecb-3646-4836-b815-6ef146b19137","Type":"ContainerDied","Data":"debfff4f6df9d1ec48f18fd53cb1e6db3249e2872158bbf0fdafe9e2c2141174"} Sep 07 10:30:02.818523 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.818503 2565 generic.go:356] "Generic (PLEG): container finished" podID="bdb34e07-90de-4fdb-ab28-a6902bf866d1" containerID="719aada5c9631a5e00291e9eb8554c439b5dfabd0e6c4a559348f4a7077b815a" exitCode=0 Sep 07 10:30:02.818620 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:02.818583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" event={"ID":"bdb34e07-90de-4fdb-ab28-a6902bf866d1","Type":"ContainerDied","Data":"719aada5c9631a5e00291e9eb8554c439b5dfabd0e6c4a559348f4a7077b815a"} Sep 07 10:30:03.975681 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:03.975653 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" Sep 07 10:30:04.020397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.020360 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" Sep 07 10:30:04.031687 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.031661 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" Sep 07 10:30:04.831239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.831097 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" Sep 07 10:30:04.831239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.831148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62" event={"ID":"080e0ecb-3646-4836-b815-6ef146b19137","Type":"ContainerDied","Data":"2c1b22cdc07784a3abf6a67a6f2f042661dc469c3d59ada7674e342230a28a3e"} Sep 07 10:30:04.831239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.831221 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c1b22cdc07784a3abf6a67a6f2f042661dc469c3d59ada7674e342230a28a3e" Sep 07 10:30:04.832589 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.832572 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" Sep 07 10:30:04.832706 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.832613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z" event={"ID":"bdb34e07-90de-4fdb-ab28-a6902bf866d1","Type":"ContainerDied","Data":"6ece45781806638e34b5242d603355436f621790f9f84c86d970adc7f46e86c3"} Sep 07 10:30:04.832706 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.832650 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ece45781806638e34b5242d603355436f621790f9f84c86d970adc7f46e86c3" Sep 07 10:30:04.834039 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.834021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" event={"ID":"2e6453be-7d00-45d5-bac2-37aa0c654c37","Type":"ContainerDied","Data":"9d92e9aee094bdebddc48ee24e4c90b6e65069258d4621ae1a717bd541b3f4d5"} Sep 07 10:30:04.834126 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.834051 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d92e9aee094bdebddc48ee24e4c90b6e65069258d4621ae1a717bd541b3f4d5" Sep 07 10:30:04.834126 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:04.834050 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq" Sep 07 10:30:05.053989 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.053957 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv"] Sep 07 10:30:05.054605 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054585 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb34e07-90de-4fdb-ab28-a6902bf866d1" containerName="cleanup" Sep 07 10:30:05.054660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054610 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb34e07-90de-4fdb-ab28-a6902bf866d1" containerName="cleanup" Sep 07 10:30:05.054660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054630 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080e0ecb-3646-4836-b815-6ef146b19137" containerName="cleanup" Sep 07 10:30:05.054660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054640 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="080e0ecb-3646-4836-b815-6ef146b19137" containerName="cleanup" Sep 07 10:30:05.054660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054649 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6453be-7d00-45d5-bac2-37aa0c654c37" containerName="cleanup" Sep 07 10:30:05.054660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054658 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6453be-7d00-45d5-bac2-37aa0c654c37" containerName="cleanup" Sep 07 10:30:05.054825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054748 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6453be-7d00-45d5-bac2-37aa0c654c37" containerName="cleanup" Sep 07 10:30:05.054825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054760 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb34e07-90de-4fdb-ab28-a6902bf866d1" containerName="cleanup" Sep 07 10:30:05.054825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.054770 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="080e0ecb-3646-4836-b815-6ef146b19137" containerName="cleanup" Sep 07 10:30:05.057944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.057924 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.060477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.060460 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-319d4c-openshift-default-dockercfg-d7mwd\"" Sep 07 10:30:05.068937 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.068911 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv"] Sep 07 10:30:05.172940 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.172834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-envoy\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.172940 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.172887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-credential-socket\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.172940 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.172935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-certs\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.173647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.172976 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7vh\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-kube-api-access-gp7vh\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.173647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.173029 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-podinfo\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.173647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.173080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1e1f-6459-4f77-8d03-d448a6e90964-istiod-ca-cert\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.173647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.173282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-token\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.173647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.173317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-data\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.173647 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.173384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-socket\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.273928 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.273887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-credential-socket\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.273939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-certs\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.273962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7vh\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-kube-api-access-gp7vh\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.273988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-podinfo\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1e1f-6459-4f77-8d03-d448a6e90964-istiod-ca-cert\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-token\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274132 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-data\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-socket\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-envoy\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-certs\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-data\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-credential-socket\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274748 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274662 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-socket\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.274869 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.274848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1e1f-6459-4f77-8d03-d448a6e90964-istiod-ca-cert\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.276470 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.276450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-envoy\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.276717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.276698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-podinfo\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.287665 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.287642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7vh\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-kube-api-access-gp7vh\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.287807 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.287778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-token\") pod \"e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.369899 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.369859 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:05.648407 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.648380 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv"] Sep 07 10:30:05.649940 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:30:05.649909 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bd1e1f_6459_4f77_8d03_d448a6e90964.slice/crio-a300bde6cfa90d898fa4c905d5e11bb497bb79c42d537b68691463c77adf3768 WatchSource:0}: Error finding container a300bde6cfa90d898fa4c905d5e11bb497bb79c42d537b68691463c77adf3768: Status 404 returned error can't find the container with id a300bde6cfa90d898fa4c905d5e11bb497bb79c42d537b68691463c77adf3768 Sep 07 10:30:05.652379 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.652343 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:05.652470 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.652413 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:05.652470 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.652441 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:05.843386 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.843352 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" event={"ID":"77bd1e1f-6459-4f77-8d03-d448a6e90964","Type":"ContainerStarted","Data":"2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521"} Sep 07 10:30:05.843595 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.843471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" event={"ID":"77bd1e1f-6459-4f77-8d03-d448a6e90964","Type":"ContainerStarted","Data":"a300bde6cfa90d898fa4c905d5e11bb497bb79c42d537b68691463c77adf3768"} Sep 07 10:30:05.863973 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:05.863910 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" podStartSLOduration=0.863889811 podStartE2EDuration="863.889811ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:05.861718951 +0000 UTC m=+1127.847999290" watchObservedRunningTime="2026-09-07 10:30:05.863889811 +0000 UTC m=+1127.850170146" Sep 07 10:30:06.370824 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:06.370788 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:06.375925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:06.375900 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:06.850099 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:06.850066 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:06.850973 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:06.850955 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:25.971493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:25.971459 2565 generic.go:356] "Generic (PLEG): container finished" podID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerID="2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521" exitCode=0 Sep 07 10:30:25.971912 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:25.971532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" event={"ID":"77bd1e1f-6459-4f77-8d03-d448a6e90964","Type":"ContainerDied","Data":"2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521"} Sep 07 10:30:25.971985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:25.971967 2565 scope.go:117] "RemoveContainer" containerID="2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521" Sep 07 10:30:25.973213 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:25.973188 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:25.973285 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:25.973252 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:25.973327 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:25.973282 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:26.979331 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:26.979294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" event={"ID":"77bd1e1f-6459-4f77-8d03-d448a6e90964","Type":"ContainerStarted","Data":"8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7"} Sep 07 10:30:27.370607 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:27.370518 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:27.370762 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:27.370744 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 07 10:30:27.370823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:27.370803 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 07 10:30:28.371088 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:28.371055 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 07 10:30:28.371592 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:28.371113 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 07 10:30:29.371216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:29.371144 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 07 10:30:29.371616 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:29.371254 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 07 10:30:30.370640 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:30.370608 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 07 10:30:30.370829 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:30.370665 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 07 10:30:31.370331 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:31.370293 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 07 10:30:31.370768 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:31.370354 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 07 10:30:32.465598 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:32.465541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:32.466182 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:32.466137 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:32.466985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:32.466962 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:30:33.395404 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.395371 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8014-140224719722080"] Sep 07 10:30:33.398309 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.398292 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:33.400658 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.400639 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qpjjb\"" Sep 07 10:30:33.404715 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.404688 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-140224719722080"] Sep 07 10:30:33.558181 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.558132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjwg\" (UniqueName: \"kubernetes.io/projected/fb555e6f-f16b-456d-b532-33af61cf4e43-kube-api-access-gcjwg\") pod \"test-curl-8014-140224719722080\" (UID: \"fb555e6f-f16b-456d-b532-33af61cf4e43\") " pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:33.659547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.659459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjwg\" (UniqueName: \"kubernetes.io/projected/fb555e6f-f16b-456d-b532-33af61cf4e43-kube-api-access-gcjwg\") pod \"test-curl-8014-140224719722080\" (UID: \"fb555e6f-f16b-456d-b532-33af61cf4e43\") " pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:33.667103 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.667077 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjwg\" (UniqueName: \"kubernetes.io/projected/fb555e6f-f16b-456d-b532-33af61cf4e43-kube-api-access-gcjwg\") pod \"test-curl-8014-140224719722080\" (UID: \"fb555e6f-f16b-456d-b532-33af61cf4e43\") " pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:33.709562 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.709534 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:33.980860 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.980831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-140224719722080"] Sep 07 10:30:33.982408 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:30:33.982380 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb555e6f_f16b_456d_b532_33af61cf4e43.slice/crio-fb0aa298a7679c3621518c9f3614e1b6104e9f4f9c02316d65779f3e2ebc49b1 WatchSource:0}: Error finding container fb0aa298a7679c3621518c9f3614e1b6104e9f4f9c02316d65779f3e2ebc49b1: Status 404 returned error can't find the container with id fb0aa298a7679c3621518c9f3614e1b6104e9f4f9c02316d65779f3e2ebc49b1 Sep 07 10:30:33.983702 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:33.983687 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:30:34.021251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:34.021221 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-140224719722080" event={"ID":"fb555e6f-f16b-456d-b532-33af61cf4e43","Type":"ContainerStarted","Data":"fb0aa298a7679c3621518c9f3614e1b6104e9f4f9c02316d65779f3e2ebc49b1"} Sep 07 10:30:36.036753 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:36.036715 2565 generic.go:356] "Generic (PLEG): container finished" podID="fb555e6f-f16b-456d-b532-33af61cf4e43" containerID="f48af4d78d75270d968666379bba4a63c4c8629a09054917122a39ee0e778607" exitCode=0 Sep 07 10:30:36.037193 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:36.036809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-140224719722080" event={"ID":"fb555e6f-f16b-456d-b532-33af61cf4e43","Type":"ContainerDied","Data":"f48af4d78d75270d968666379bba4a63c4c8629a09054917122a39ee0e778607"} Sep 07 10:30:37.172546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:37.172524 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:37.206096 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:37.206070 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-8014-140224719722080_fb555e6f-f16b-456d-b532-33af61cf4e43/test-curl-8014-140224719722080/0.log" Sep 07 10:30:37.228680 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:37.228651 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8014-140224719722080"] Sep 07 10:30:37.230815 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:37.230796 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8014-140224719722080"] Sep 07 10:30:37.297731 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:37.297658 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcjwg\" (UniqueName: \"kubernetes.io/projected/fb555e6f-f16b-456d-b532-33af61cf4e43-kube-api-access-gcjwg\") pod \"fb555e6f-f16b-456d-b532-33af61cf4e43\" (UID: \"fb555e6f-f16b-456d-b532-33af61cf4e43\") " Sep 07 10:30:37.299991 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:37.299969 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb555e6f-f16b-456d-b532-33af61cf4e43-kube-api-access-gcjwg" (OuterVolumeSpecName: "kube-api-access-gcjwg") pod "fb555e6f-f16b-456d-b532-33af61cf4e43" (UID: "fb555e6f-f16b-456d-b532-33af61cf4e43"). InnerVolumeSpecName "kube-api-access-gcjwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:30:37.399110 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:37.399077 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcjwg\" (UniqueName: \"kubernetes.io/projected/fb555e6f-f16b-456d-b532-33af61cf4e43-kube-api-access-gcjwg\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:30:38.049739 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:38.049709 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0aa298a7679c3621518c9f3614e1b6104e9f4f9c02316d65779f3e2ebc49b1" Sep 07 10:30:38.049739 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:38.049725 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:38.058969 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:30:38.058943 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8014-140224719722080\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="fb555e6f-f16b-456d-b532-33af61cf4e43" pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:38.543174 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:30:38.543133 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8014-140224719722080\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="fb555e6f-f16b-456d-b532-33af61cf4e43" pod="opendatahub/test-curl-8014-140224719722080" Sep 07 10:30:38.544483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:38.544464 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb555e6f-f16b-456d-b532-33af61cf4e43" path="/var/lib/kubelet/pods/fb555e6f-f16b-456d-b532-33af61cf4e43/volumes" Sep 07 10:30:47.045079 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.044995 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j"] Sep 07 10:30:47.045661 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.045446 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb555e6f-f16b-456d-b532-33af61cf4e43" containerName="test-curl-8014-140224719722080" Sep 07 10:30:47.045661 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.045458 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb555e6f-f16b-456d-b532-33af61cf4e43" containerName="test-curl-8014-140224719722080" Sep 07 10:30:47.045661 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.045567 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb555e6f-f16b-456d-b532-33af61cf4e43" containerName="test-curl-8014-140224719722080" Sep 07 10:30:47.048702 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.048685 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.051653 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.051637 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f957f836-openshift-default-dockercfg-gq2t9\"" Sep 07 10:30:47.067555 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.067535 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j"] Sep 07 10:30:47.089568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-socket\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.089686 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-certs\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.089686 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089593 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-credential-socket\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.089686 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-podinfo\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.089806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-data\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.089806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-envoy\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.089806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48a67f-86ba-4247-8a30-b8b138d60854-istiod-ca-cert\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.089806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089804 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-token\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.089949 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.089824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddph\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-kube-api-access-gddph\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191126 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-podinfo\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191343 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-data\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191343 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-envoy\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191343 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48a67f-86ba-4247-8a30-b8b138d60854-istiod-ca-cert\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191343 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-token\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191343 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddph\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-kube-api-access-gddph\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191620 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-socket\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191620 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-certs\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191620 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-credential-socket\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191620 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-data\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191779 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-socket\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191783 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-certs\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.191836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.191801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-credential-socket\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.192095 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.192077 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48a67f-86ba-4247-8a30-b8b138d60854-istiod-ca-cert\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.194126 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.194109 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-envoy\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.194290 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.194274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-podinfo\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.199542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.199521 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-token\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.199729 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.199709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddph\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-kube-api-access-gddph\") pod \"e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.358993 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.358897 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:47.641699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.641663 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j"] Sep 07 10:30:47.642893 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:30:47.642870 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c48a67f_86ba_4247_8a30_b8b138d60854.slice/crio-e3d1fa38c66c42af8ddca5c175c0f7d907a41e28ac7e1d12a388d3ab42a6b71e WatchSource:0}: Error finding container e3d1fa38c66c42af8ddca5c175c0f7d907a41e28ac7e1d12a388d3ab42a6b71e: Status 404 returned error can't find the container with id e3d1fa38c66c42af8ddca5c175c0f7d907a41e28ac7e1d12a388d3ab42a6b71e Sep 07 10:30:47.645459 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.645413 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:47.645544 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.645507 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:47.645593 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:47.645556 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:30:48.108541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:48.108508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" event={"ID":"7c48a67f-86ba-4247-8a30-b8b138d60854","Type":"ContainerStarted","Data":"c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0"} Sep 07 10:30:48.108987 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:48.108561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" event={"ID":"7c48a67f-86ba-4247-8a30-b8b138d60854","Type":"ContainerStarted","Data":"e3d1fa38c66c42af8ddca5c175c0f7d907a41e28ac7e1d12a388d3ab42a6b71e"} Sep 07 10:30:48.131580 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:48.131522 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" podStartSLOduration=1.13150662 podStartE2EDuration="1.13150662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:48.129291173 +0000 UTC m=+1170.115571510" watchObservedRunningTime="2026-09-07 10:30:48.13150662 +0000 UTC m=+1170.117786955" Sep 07 10:30:48.359921 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:48.359834 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:48.364874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:48.364848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:49.114476 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:49.114444 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:30:49.115396 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:30:49.115372 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:31:09.251288 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:09.251253 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerID="c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0" exitCode=0 Sep 07 10:31:09.251770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:09.251328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" event={"ID":"7c48a67f-86ba-4247-8a30-b8b138d60854","Type":"ContainerDied","Data":"c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0"} Sep 07 10:31:09.251770 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:09.251726 2565 scope.go:117] "RemoveContainer" containerID="c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0" Sep 07 10:31:09.252836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:09.252809 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:31:09.252908 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:09.252894 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:31:09.252946 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:09.252929 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:31:10.259112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:10.259073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" event={"ID":"7c48a67f-86ba-4247-8a30-b8b138d60854","Type":"ContainerStarted","Data":"7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8"} Sep 07 10:31:10.359543 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:10.359503 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:31:10.359753 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:10.359729 2565 patch_prober.go:28] interesting pod/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 07 10:31:10.359809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:10.359789 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 07 10:31:11.360005 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:11.359965 2565 patch_prober.go:28] interesting pod/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 07 10:31:11.360431 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:11.360030 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 07 10:31:12.360357 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:12.360318 2565 patch_prober.go:28] interesting pod/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 07 10:31:12.360774 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:12.360380 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 07 10:31:13.359852 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:13.359811 2565 patch_prober.go:28] interesting pod/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 07 10:31:13.360051 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:13.359883 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 07 10:31:14.360093 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:14.360058 2565 patch_prober.go:28] interesting pod/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 07 10:31:14.360519 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:14.360112 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 07 10:31:15.359905 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:15.359863 2565 patch_prober.go:28] interesting pod/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 07 10:31:15.360104 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:15.359921 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 07 10:31:16.363909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:16.363879 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:31:16.364318 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:16.364276 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:31:16.365040 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:16.365023 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:31:18.592966 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:18.592929 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:31:18.595901 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:18.595870 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:31:18.671251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:18.671218 2565 scope.go:117] "RemoveContainer" containerID="975c9127d60f069992e76165b1fded2b2bfc38bfd7b0b9d5fcbcdc898ea52465" Sep 07 10:31:28.780501 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.780467 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6"] Sep 07 10:31:28.785678 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.785655 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.789403 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.789379 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"kube-root-ca.crt\"" Sep 07 10:31:28.789704 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.789680 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"auth-test-model-93e41e-kserve-self-signed-certs\"" Sep 07 10:31:28.789813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.789725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"default-dockercfg-pwz5t\"" Sep 07 10:31:28.789813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.789725 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"openshift-service-ca.crt\"" Sep 07 10:31:28.793728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.793701 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6"] Sep 07 10:31:28.843249 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.843211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-home\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.843449 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.843266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-dshm\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.843449 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.843359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tls-certs\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.843449 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.843407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-model-cache\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.843593 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.843487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fvm\" (UniqueName: \"kubernetes.io/projected/a3290f1c-0e8a-4b51-8d10-922a10178ea4-kube-api-access-86fvm\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.843593 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.843536 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tmp-dir\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.944418 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tmp-dir\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.944619 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-home\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.944619 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-dshm\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.944619 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944559 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tls-certs\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.944619 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-model-cache\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.944839 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fvm\" (UniqueName: \"kubernetes.io/projected/a3290f1c-0e8a-4b51-8d10-922a10178ea4-kube-api-access-86fvm\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.944839 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tmp-dir\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.944944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-home\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.945004 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.944951 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-model-cache\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.946938 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.946914 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-dshm\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.947085 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.947070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tls-certs\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:28.952375 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:28.952356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fvm\" (UniqueName: \"kubernetes.io/projected/a3290f1c-0e8a-4b51-8d10-922a10178ea4-kube-api-access-86fvm\") pod \"auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:29.098861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:29.098754 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:29.378355 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:29.378332 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6"] Sep 07 10:31:29.380551 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:31:29.380524 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3290f1c_0e8a_4b51_8d10_922a10178ea4.slice/crio-6e795bb1adc361280b2a0c0adcefc77fb5aab6f7266d340f66d23a180f58da70 WatchSource:0}: Error finding container 6e795bb1adc361280b2a0c0adcefc77fb5aab6f7266d340f66d23a180f58da70: Status 404 returned error can't find the container with id 6e795bb1adc361280b2a0c0adcefc77fb5aab6f7266d340f66d23a180f58da70 Sep 07 10:31:30.384743 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:30.384697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" event={"ID":"a3290f1c-0e8a-4b51-8d10-922a10178ea4","Type":"ContainerStarted","Data":"9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d"} Sep 07 10:31:30.384743 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:30.384747 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" event={"ID":"a3290f1c-0e8a-4b51-8d10-922a10178ea4","Type":"ContainerStarted","Data":"6e795bb1adc361280b2a0c0adcefc77fb5aab6f7266d340f66d23a180f58da70"} Sep 07 10:31:30.401806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:30.401760 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" podStartSLOduration=2.401744869 podStartE2EDuration="2.401744869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:31:30.400916717 +0000 UTC m=+1212.387197052" watchObservedRunningTime="2026-09-07 10:31:30.401744869 +0000 UTC m=+1212.388025203" Sep 07 10:31:39.099187 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:39.099106 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:39.099637 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:39.099209 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:39.111878 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:39.111851 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:31:39.451850 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:31:39.451759 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:32:18.693251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:18.693154 2565 scope.go:117] "RemoveContainer" containerID="ed050a72522192454f9f2226d8fe969c5afe97d623f6e0db3ee3cb367a49bd62" Sep 07 10:32:18.702141 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:18.702121 2565 scope.go:117] "RemoveContainer" containerID="a8905c5b2bef80dc2c6230ffc9560360a7126a352f7d8c0898504fe1eea68c46" Sep 07 10:32:18.710794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:18.710773 2565 scope.go:117] "RemoveContainer" containerID="8c08321614fefda521dfd4bceb6c5ca140aab4573dd3fdc81f6e536ade987ff4" Sep 07 10:32:30.654372 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.654336 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw"] Sep 07 10:32:30.659869 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.659849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" Sep 07 10:32:30.662494 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.662462 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:32:30.662731 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.662712 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:32:30.664767 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.664736 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw"] Sep 07 10:32:30.680361 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.680327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31bdae5-6e1d-4390-9f11-61f0411d5b75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f957f836-dl7nw\" (UID: \"d31bdae5-6e1d-4390-9f11-61f0411d5b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" Sep 07 10:32:30.781691 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.781646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31bdae5-6e1d-4390-9f11-61f0411d5b75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f957f836-dl7nw\" (UID: \"d31bdae5-6e1d-4390-9f11-61f0411d5b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" Sep 07 10:32:30.782244 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.782223 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31bdae5-6e1d-4390-9f11-61f0411d5b75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f957f836-dl7nw\" (UID: \"d31bdae5-6e1d-4390-9f11-61f0411d5b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" Sep 07 10:32:30.975318 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:30.975282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" Sep 07 10:32:31.264334 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:31.264302 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw"] Sep 07 10:32:31.265912 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:32:31.265885 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31bdae5_6e1d_4390_9f11_61f0411d5b75.slice/crio-d5bd301c3524148b13b6112d9af25645a049e4784d04b2a89efab0e215a028fa WatchSource:0}: Error finding container d5bd301c3524148b13b6112d9af25645a049e4784d04b2a89efab0e215a028fa: Status 404 returned error can't find the container with id d5bd301c3524148b13b6112d9af25645a049e4784d04b2a89efab0e215a028fa Sep 07 10:32:31.769763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:31.769732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" event={"ID":"d31bdae5-6e1d-4390-9f11-61f0411d5b75","Type":"ContainerStarted","Data":"be0937c2760057ee2f52c68e806e2299154001432eb0b21115f58450418c1762"} Sep 07 10:32:31.770197 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:31.769781 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" event={"ID":"d31bdae5-6e1d-4390-9f11-61f0411d5b75","Type":"ContainerStarted","Data":"d5bd301c3524148b13b6112d9af25645a049e4784d04b2a89efab0e215a028fa"} Sep 07 10:32:31.790065 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:31.790010 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" podStartSLOduration=1.7899918719999999 podStartE2EDuration="1.789991872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:32:31.787666308 +0000 UTC m=+1273.773946642" watchObservedRunningTime="2026-09-07 10:32:31.789991872 +0000 UTC m=+1273.776272206" Sep 07 10:32:32.777645 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:32.777612 2565 generic.go:356] "Generic (PLEG): container finished" podID="d31bdae5-6e1d-4390-9f11-61f0411d5b75" containerID="be0937c2760057ee2f52c68e806e2299154001432eb0b21115f58450418c1762" exitCode=0 Sep 07 10:32:32.778081 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:32.777701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" event={"ID":"d31bdae5-6e1d-4390-9f11-61f0411d5b75","Type":"ContainerDied","Data":"be0937c2760057ee2f52c68e806e2299154001432eb0b21115f58450418c1762"} Sep 07 10:32:33.927440 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:33.927408 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" Sep 07 10:32:34.024817 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:34.024781 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31bdae5-6e1d-4390-9f11-61f0411d5b75-maas-api-service-ca\") pod \"d31bdae5-6e1d-4390-9f11-61f0411d5b75\" (UID: \"d31bdae5-6e1d-4390-9f11-61f0411d5b75\") " Sep 07 10:32:34.025252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:34.025225 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31bdae5-6e1d-4390-9f11-61f0411d5b75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d31bdae5-6e1d-4390-9f11-61f0411d5b75" (UID: "d31bdae5-6e1d-4390-9f11-61f0411d5b75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:32:34.126321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:34.126224 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31bdae5-6e1d-4390-9f11-61f0411d5b75-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:32:34.790688 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:34.790655 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" Sep 07 10:32:34.790874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:34.790697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw" event={"ID":"d31bdae5-6e1d-4390-9f11-61f0411d5b75","Type":"ContainerDied","Data":"d5bd301c3524148b13b6112d9af25645a049e4784d04b2a89efab0e215a028fa"} Sep 07 10:32:34.790874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:34.790760 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5bd301c3524148b13b6112d9af25645a049e4784d04b2a89efab0e215a028fa" Sep 07 10:32:46.472961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:46.472923 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw"] Sep 07 10:32:46.475245 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:46.475222 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f957f836-dl7nw"] Sep 07 10:32:46.552184 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:32:46.549802 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d31bdae5-6e1d-4390-9f11-61f0411d5b75" path="/var/lib/kubelet/pods/d31bdae5-6e1d-4390-9f11-61f0411d5b75/volumes" Sep 07 10:33:18.743352 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:18.743318 2565 scope.go:117] "RemoveContainer" containerID="f809bbc3f775a56279f3a6c896452eeddeead5912c32e66d0eff55835baf4328" Sep 07 10:33:37.715196 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:37.715142 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j"] Sep 07 10:33:37.715625 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:37.715430 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" containerID="cri-o://7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8" gracePeriod=30 Sep 07 10:33:42.973845 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:42.973776 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:33:43.020803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.020771 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-certs\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.020969 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.020910 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gddph\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-kube-api-access-gddph\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.021028 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.020956 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-socket\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.021028 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.020996 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-podinfo\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021137 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-envoy\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021218 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-token\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021256 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-data\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021279 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021286 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-credential-socket\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021369 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48a67f-86ba-4247-8a30-b8b138d60854-istiod-ca-cert\") pod \"7c48a67f-86ba-4247-8a30-b8b138d60854\" (UID: \"7c48a67f-86ba-4247-8a30-b8b138d60854\") " Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021681 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021791 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021814 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.021874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021832 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.022376 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.021900 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:33:43.022376 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.022070 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c48a67f-86ba-4247-8a30-b8b138d60854-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:33:43.023944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.023919 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-kube-api-access-gddph" (OuterVolumeSpecName: "kube-api-access-gddph") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "kube-api-access-gddph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:33:43.023944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.023926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:33:43.024140 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.024074 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:33:43.024341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.024148 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c48a67f-86ba-4247-8a30-b8b138d60854" (UID: "7c48a67f-86ba-4247-8a30-b8b138d60854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:33:43.122699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.122663 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48a67f-86ba-4247-8a30-b8b138d60854-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.122699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.122693 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.122699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.122702 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gddph\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-kube-api-access-gddph\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.122931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.122710 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.122931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.122719 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.122931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.122726 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48a67f-86ba-4247-8a30-b8b138d60854-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.122931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.122733 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48a67f-86ba-4247-8a30-b8b138d60854-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:33:43.197452 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.197420 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerID="7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8" exitCode=0 Sep 07 10:33:43.197628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.197472 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" Sep 07 10:33:43.197628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.197498 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" event={"ID":"7c48a67f-86ba-4247-8a30-b8b138d60854","Type":"ContainerDied","Data":"7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8"} Sep 07 10:33:43.197628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.197551 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j" event={"ID":"7c48a67f-86ba-4247-8a30-b8b138d60854","Type":"ContainerDied","Data":"e3d1fa38c66c42af8ddca5c175c0f7d907a41e28ac7e1d12a388d3ab42a6b71e"} Sep 07 10:33:43.197628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.197574 2565 scope.go:117] "RemoveContainer" containerID="7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8" Sep 07 10:33:43.207879 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.207861 2565 scope.go:117] "RemoveContainer" containerID="c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0" Sep 07 10:33:43.216304 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.216287 2565 scope.go:117] "RemoveContainer" containerID="7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8" Sep 07 10:33:43.216559 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:33:43.216542 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8\": container with ID starting with 7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8 not found: ID does not exist" containerID="7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8" Sep 07 10:33:43.216604 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.216567 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8"} err="failed to get container status \"7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8\": rpc error: code = NotFound desc = could not find container \"7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8\": container with ID starting with 7a227e3e8a76e32b56e6f9db10794451fe95b3fb1e7800838a99788aefd2d6e8 not found: ID does not exist" Sep 07 10:33:43.216604 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.216581 2565 scope.go:117] "RemoveContainer" containerID="c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0" Sep 07 10:33:43.216809 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:33:43.216792 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0\": container with ID starting with c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0 not found: ID does not exist" containerID="c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0" Sep 07 10:33:43.216855 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.216815 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0"} err="failed to get container status \"c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0\": rpc error: code = NotFound desc = could not find container \"c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0\": container with ID starting with c7791f90a856f7a0d916f9838f9aac7626941f7a5f412b12a7b47526ba02d7d0 not found: ID does not exist" Sep 07 10:33:43.222815 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.222789 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j"] Sep 07 10:33:43.231805 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:43.231748 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f957f836-openshift-default-758f48b9b4-rlp8j"] Sep 07 10:33:44.544178 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:44.544135 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" path="/var/lib/kubelet/pods/7c48a67f-86ba-4247-8a30-b8b138d60854/volumes" Sep 07 10:33:52.602011 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:33:52.601964 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6"] Sep 07 10:34:07.623213 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.623171 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" podUID="a3290f1c-0e8a-4b51-8d10-922a10178ea4" containerName="main" containerID="cri-o://9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d" gracePeriod=45 Sep 07 10:34:07.879245 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.879181 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:34:07.975816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.975783 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-home\") pod \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " Sep 07 10:34:07.976003 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.975827 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-model-cache\") pod \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " Sep 07 10:34:07.976003 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.975870 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-dshm\") pod \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " Sep 07 10:34:07.976003 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.975920 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tmp-dir\") pod \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " Sep 07 10:34:07.976003 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.975971 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tls-certs\") pod \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " Sep 07 10:34:07.976274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.976010 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86fvm\" (UniqueName: \"kubernetes.io/projected/a3290f1c-0e8a-4b51-8d10-922a10178ea4-kube-api-access-86fvm\") pod \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\" (UID: \"a3290f1c-0e8a-4b51-8d10-922a10178ea4\") " Sep 07 10:34:07.976274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.976100 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-home" (OuterVolumeSpecName: "home") pod "a3290f1c-0e8a-4b51-8d10-922a10178ea4" (UID: "a3290f1c-0e8a-4b51-8d10-922a10178ea4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:34:07.976274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.976202 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3290f1c-0e8a-4b51-8d10-922a10178ea4" (UID: "a3290f1c-0e8a-4b51-8d10-922a10178ea4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:34:07.976408 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.976297 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3290f1c-0e8a-4b51-8d10-922a10178ea4" (UID: "a3290f1c-0e8a-4b51-8d10-922a10178ea4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:34:07.976408 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.976400 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:34:07.976475 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.976421 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:34:07.976475 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.976435 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:34:07.978428 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.978398 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-dshm" (OuterVolumeSpecName: "dshm") pod "a3290f1c-0e8a-4b51-8d10-922a10178ea4" (UID: "a3290f1c-0e8a-4b51-8d10-922a10178ea4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:34:07.978428 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.978409 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3290f1c-0e8a-4b51-8d10-922a10178ea4-kube-api-access-86fvm" (OuterVolumeSpecName: "kube-api-access-86fvm") pod "a3290f1c-0e8a-4b51-8d10-922a10178ea4" (UID: "a3290f1c-0e8a-4b51-8d10-922a10178ea4"). InnerVolumeSpecName "kube-api-access-86fvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:34:07.978611 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:07.978500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3290f1c-0e8a-4b51-8d10-922a10178ea4" (UID: "a3290f1c-0e8a-4b51-8d10-922a10178ea4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:34:08.077593 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.077561 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3290f1c-0e8a-4b51-8d10-922a10178ea4-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:34:08.077593 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.077590 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86fvm\" (UniqueName: \"kubernetes.io/projected/a3290f1c-0e8a-4b51-8d10-922a10178ea4-kube-api-access-86fvm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:34:08.077593 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.077600 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3290f1c-0e8a-4b51-8d10-922a10178ea4-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:34:08.346417 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.346385 2565 generic.go:356] "Generic (PLEG): container finished" podID="a3290f1c-0e8a-4b51-8d10-922a10178ea4" containerID="9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d" exitCode=0 Sep 07 10:34:08.346634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.346424 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" Sep 07 10:34:08.346634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.346479 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" event={"ID":"a3290f1c-0e8a-4b51-8d10-922a10178ea4","Type":"ContainerDied","Data":"9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d"} Sep 07 10:34:08.346634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.346525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6" event={"ID":"a3290f1c-0e8a-4b51-8d10-922a10178ea4","Type":"ContainerDied","Data":"6e795bb1adc361280b2a0c0adcefc77fb5aab6f7266d340f66d23a180f58da70"} Sep 07 10:34:08.346634 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.346548 2565 scope.go:117] "RemoveContainer" containerID="9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d" Sep 07 10:34:08.356607 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.356588 2565 scope.go:117] "RemoveContainer" containerID="9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d" Sep 07 10:34:08.356869 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:34:08.356848 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d\": container with ID starting with 9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d not found: ID does not exist" containerID="9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d" Sep 07 10:34:08.356923 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.356880 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d"} err="failed to get container status \"9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d\": rpc error: code = NotFound desc = could not find container \"9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d\": container with ID starting with 9b451c54c33071a7d404d446605e1a7124eccbd6efd7e8b1cf6281a04907fa7d not found: ID does not exist" Sep 07 10:34:08.368098 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.368077 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6"] Sep 07 10:34:08.372219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.372199 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/auth-test-model-93e41e-kserve-5b7bb7ff5-ckgd6"] Sep 07 10:34:08.544068 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:08.544035 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3290f1c-0e8a-4b51-8d10-922a10178ea4" path="/var/lib/kubelet/pods/a3290f1c-0e8a-4b51-8d10-922a10178ea4/volumes" Sep 07 10:34:18.766448 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:18.766413 2565 scope.go:117] "RemoveContainer" containerID="461ba9afb4dfc7b052a61fe4a265b7b91b485cd6384fc4c6497d261a83bf458b" Sep 07 10:34:18.775217 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:18.775198 2565 scope.go:117] "RemoveContainer" containerID="5e90cc46c4de5195f5f3be39fe2a33bab4f00cbae418117949dd7336bf890de6" Sep 07 10:34:31.640802 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.640763 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95"] Sep 07 10:34:31.641274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641182 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" Sep 07 10:34:31.641274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641194 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" Sep 07 10:34:31.641274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641216 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3290f1c-0e8a-4b51-8d10-922a10178ea4" containerName="main" Sep 07 10:34:31.641274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641222 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3290f1c-0e8a-4b51-8d10-922a10178ea4" containerName="main" Sep 07 10:34:31.641274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641228 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31bdae5-6e1d-4390-9f11-61f0411d5b75" containerName="revoke-keys" Sep 07 10:34:31.641274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641234 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31bdae5-6e1d-4390-9f11-61f0411d5b75" containerName="revoke-keys" Sep 07 10:34:31.641486 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641305 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" Sep 07 10:34:31.641486 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641312 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3290f1c-0e8a-4b51-8d10-922a10178ea4" containerName="main" Sep 07 10:34:31.641486 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641321 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31bdae5-6e1d-4390-9f11-61f0411d5b75" containerName="revoke-keys" Sep 07 10:34:31.641486 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.641329 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" Sep 07 10:34:31.646247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.646218 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.648853 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.648833 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fbea94-openshift-default-dockercfg-9whtc\"" Sep 07 10:34:31.657528 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.657502 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95"] Sep 07 10:34:31.727102 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-certs\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.727102 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-istio-token\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.727360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7237ca7-155b-4512-9708-379a343ff40b-istio-podinfo\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.727360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-credential-socket\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.727360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-socket\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.727360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-envoy\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.727522 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6jv\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-kube-api-access-gb6jv\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.727522 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-data\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.727522 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.727427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7237ca7-155b-4512-9708-379a343ff40b-istiod-ca-cert\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.828485 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-socket\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.828668 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-envoy\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.828668 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6jv\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-kube-api-access-gb6jv\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.828668 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-data\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.828836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7237ca7-155b-4512-9708-379a343ff40b-istiod-ca-cert\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.828836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-certs\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.828836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-istio-token\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.828836 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7237ca7-155b-4512-9708-379a343ff40b-istio-podinfo\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.829057 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-credential-socket\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.829057 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828887 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-socket\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.829057 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.828999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-data\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.829237 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.829062 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-certs\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.829237 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.829150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-credential-socket\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.829443 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.829423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7237ca7-155b-4512-9708-379a343ff40b-istiod-ca-cert\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.830926 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.830903 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-envoy\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.831475 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.831457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7237ca7-155b-4512-9708-379a343ff40b-istio-podinfo\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.838683 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.838659 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-istio-token\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.839032 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.839012 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6jv\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-kube-api-access-gb6jv\") pod \"e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:31.958338 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:31.958248 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:32.238438 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:32.238413 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95"] Sep 07 10:34:32.239523 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:34:32.239500 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7237ca7_155b_4512_9708_379a343ff40b.slice/crio-4e9f36855aed41e143adf418ec988e044ac8824802974cbe1bd0c621f6756f6a WatchSource:0}: Error finding container 4e9f36855aed41e143adf418ec988e044ac8824802974cbe1bd0c621f6756f6a: Status 404 returned error can't find the container with id 4e9f36855aed41e143adf418ec988e044ac8824802974cbe1bd0c621f6756f6a Sep 07 10:34:32.241854 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:32.241823 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:34:32.241914 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:32.241885 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:34:32.241950 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:32.241914 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:34:32.490800 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:32.490701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" event={"ID":"f7237ca7-155b-4512-9708-379a343ff40b","Type":"ContainerStarted","Data":"3af6416c4e4179330aad034c5e45593e3eb72b547deee94dd6dd7f5a6d39ffc8"} Sep 07 10:34:32.490800 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:32.490765 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" event={"ID":"f7237ca7-155b-4512-9708-379a343ff40b","Type":"ContainerStarted","Data":"4e9f36855aed41e143adf418ec988e044ac8824802974cbe1bd0c621f6756f6a"} Sep 07 10:34:32.512292 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:32.512232 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" podStartSLOduration=1.512211033 podStartE2EDuration="1.512211033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:34:32.510042514 +0000 UTC m=+1394.496322849" watchObservedRunningTime="2026-09-07 10:34:32.512211033 +0000 UTC m=+1394.498491367" Sep 07 10:34:32.958556 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:32.958462 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:33.965176 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:33.965131 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:34.501887 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:34.501857 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:34.502883 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:34.502863 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:47.499947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.499916 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s"] Sep 07 10:34:47.500392 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.500378 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" Sep 07 10:34:47.500392 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.500394 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c48a67f-86ba-4247-8a30-b8b138d60854" containerName="istio-proxy" Sep 07 10:34:47.503557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.503527 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.507427 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.507404 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"openshift-service-ca.crt\"" Sep 07 10:34:47.507809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.507441 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"kube-root-ca.crt\"" Sep 07 10:34:47.507930 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.507535 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"sub-test-model-93e41e-kserve-self-signed-certs\"" Sep 07 10:34:47.508274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.508102 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-93e41e\"/\"default-dockercfg-pwz5t\"" Sep 07 10:34:47.511126 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.511102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s"] Sep 07 10:34:47.589536 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.589494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-model-cache\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.589717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.589609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-home\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.589717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.589670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-tmp-dir\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.589717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.589698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-dshm\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.589840 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.589720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f1a321-96a5-4205-af95-c7d44e1cc110-tls-certs\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.589840 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.589764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfm6\" (UniqueName: \"kubernetes.io/projected/62f1a321-96a5-4205-af95-c7d44e1cc110-kube-api-access-6vfm6\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.690429 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.690386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-model-cache\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.690628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.690452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-home\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.690628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.690479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-tmp-dir\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.690628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.690495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-dshm\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.690628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.690510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f1a321-96a5-4205-af95-c7d44e1cc110-tls-certs\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.690628 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.690528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfm6\" (UniqueName: \"kubernetes.io/projected/62f1a321-96a5-4205-af95-c7d44e1cc110-kube-api-access-6vfm6\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.690872 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.690851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-model-cache\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.690938 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.690911 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-home\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.691042 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.691019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-tmp-dir\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.692961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.692942 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-dshm\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.693124 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.693110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f1a321-96a5-4205-af95-c7d44e1cc110-tls-certs\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.700203 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.700151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfm6\" (UniqueName: \"kubernetes.io/projected/62f1a321-96a5-4205-af95-c7d44e1cc110-kube-api-access-6vfm6\") pod \"sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:47.814673 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:47.814582 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:48.110278 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:48.110251 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s"] Sep 07 10:34:48.112431 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:34:48.112406 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62f1a321_96a5_4205_af95_c7d44e1cc110.slice/crio-86d01999324f423535dbc61134b8ecebac0a16109b9301c6379f1b4650a972a8 WatchSource:0}: Error finding container 86d01999324f423535dbc61134b8ecebac0a16109b9301c6379f1b4650a972a8: Status 404 returned error can't find the container with id 86d01999324f423535dbc61134b8ecebac0a16109b9301c6379f1b4650a972a8 Sep 07 10:34:48.585292 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:48.585256 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" event={"ID":"62f1a321-96a5-4205-af95-c7d44e1cc110","Type":"ContainerStarted","Data":"614ee306292599cf773d4af2fda0590bc5a9893f2a7ecca251861be1f088d975"} Sep 07 10:34:48.585703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:48.585304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" event={"ID":"62f1a321-96a5-4205-af95-c7d44e1cc110","Type":"ContainerStarted","Data":"86d01999324f423535dbc61134b8ecebac0a16109b9301c6379f1b4650a972a8"} Sep 07 10:34:48.603896 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:48.603847 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" podStartSLOduration=1.603831423 podStartE2EDuration="1.603831423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:34:48.601232794 +0000 UTC m=+1410.587513129" watchObservedRunningTime="2026-09-07 10:34:48.603831423 +0000 UTC m=+1410.590111798" Sep 07 10:34:54.625616 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:54.625581 2565 generic.go:356] "Generic (PLEG): container finished" podID="f7237ca7-155b-4512-9708-379a343ff40b" containerID="3af6416c4e4179330aad034c5e45593e3eb72b547deee94dd6dd7f5a6d39ffc8" exitCode=0 Sep 07 10:34:54.625616 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:54.625621 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" event={"ID":"f7237ca7-155b-4512-9708-379a343ff40b","Type":"ContainerDied","Data":"3af6416c4e4179330aad034c5e45593e3eb72b547deee94dd6dd7f5a6d39ffc8"} Sep 07 10:34:54.626096 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:54.626057 2565 scope.go:117] "RemoveContainer" containerID="3af6416c4e4179330aad034c5e45593e3eb72b547deee94dd6dd7f5a6d39ffc8" Sep 07 10:34:54.627287 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:54.627260 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:34:54.627344 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:54.627332 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:34:54.627378 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:54.627361 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:34:55.634345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:55.634308 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" event={"ID":"f7237ca7-155b-4512-9708-379a343ff40b","Type":"ContainerStarted","Data":"1e694408a614dcfb203a003c24213502b7ce2f7df4855312e2b2e9d182f49b0a"} Sep 07 10:34:55.958890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:55.958779 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:34:55.959082 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:55.958986 2565 patch_prober.go:28] interesting pod/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 10:34:55.959082 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:55.959071 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 10:34:56.958737 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:56.958696 2565 patch_prober.go:28] interesting pod/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 10:34:56.959191 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:56.958755 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 10:34:57.815488 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:57.815452 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:57.815696 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:57.815509 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:57.828363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:57.828337 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:57.958998 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:57.958960 2565 patch_prober.go:28] interesting pod/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 10:34:57.959455 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:57.959026 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 10:34:58.672384 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:58.672350 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:34:58.958777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:58.958681 2565 patch_prober.go:28] interesting pod/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 10:34:58.959051 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:58.959023 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 10:34:59.959666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:59.959633 2565 patch_prober.go:28] interesting pod/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 07 10:34:59.960142 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:34:59.959832 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 07 10:35:00.069183 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.068565 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2"] Sep 07 10:35:00.074864 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.074836 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.075935 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.075894 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2"] Sep 07 10:35:00.078072 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.078031 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-695a13-openshift-default-dockercfg-l2g9l\"" Sep 07 10:35:00.223904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.223877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istiod-ca-cert\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.224062 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.223923 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-certs\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.224062 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.223969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-podinfo\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.224062 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.223990 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-data\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.224062 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.224048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-credential-socket\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.224239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.224121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-socket\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.224239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.224146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-token\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.224239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.224228 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-envoy\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.224335 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.224275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggs2f\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-kube-api-access-ggs2f\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325516 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-socket\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325695 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-token\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325695 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-envoy\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325695 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggs2f\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-kube-api-access-ggs2f\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325695 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istiod-ca-cert\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325695 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-certs\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-podinfo\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-data\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-credential-socket\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.325870 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.325851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-socket\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.326083 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.326054 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-credential-socket\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.326402 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.326366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-data\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.326464 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.326440 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istiod-ca-cert\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.326654 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.326623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-certs\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.328495 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.328475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-envoy\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.328815 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.328799 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-podinfo\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.334818 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.334773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggs2f\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-kube-api-access-ggs2f\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.340618 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.340589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-token\") pod \"e2e-api-b-695a13-openshift-default-984d8f756-zhng2\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.387564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.387499 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:00.686826 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.686803 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2"] Sep 07 10:35:00.688962 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:35:00.688921 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd2dbaa_3d01_4300_a66f_900d60e712ef.slice/crio-ee63d5f5660a8ab2f5af4f49ba2c431642cb77a5f1e063161e12865b01a24235 WatchSource:0}: Error finding container ee63d5f5660a8ab2f5af4f49ba2c431642cb77a5f1e063161e12865b01a24235: Status 404 returned error can't find the container with id ee63d5f5660a8ab2f5af4f49ba2c431642cb77a5f1e063161e12865b01a24235 Sep 07 10:35:00.691043 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.691011 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:35:00.691143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.691087 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:35:00.691143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:00.691130 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:35:01.202800 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:01.202762 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:35:01.203307 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:01.203273 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:35:01.204251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:01.204209 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:35:01.680302 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:01.680263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" event={"ID":"0bd2dbaa-3d01-4300-a66f-900d60e712ef","Type":"ContainerStarted","Data":"116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da"} Sep 07 10:35:01.680506 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:01.680363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" event={"ID":"0bd2dbaa-3d01-4300-a66f-900d60e712ef","Type":"ContainerStarted","Data":"ee63d5f5660a8ab2f5af4f49ba2c431642cb77a5f1e063161e12865b01a24235"} Sep 07 10:35:01.699733 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:01.699679 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" podStartSLOduration=1.699661035 podStartE2EDuration="1.699661035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:01.697717357 +0000 UTC m=+1423.683997691" watchObservedRunningTime="2026-09-07 10:35:01.699661035 +0000 UTC m=+1423.685941446" Sep 07 10:35:02.388385 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:02.388345 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:02.393666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:02.393628 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:02.686498 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:02.686411 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:02.687199 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:02.687182 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:18.813484 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:18.813407 2565 scope.go:117] "RemoveContainer" containerID="b4426a6d4f465f6f96cc36b4fc9be5a8ef81f33fbf7ed4321a02c7e27306ca82" Sep 07 10:35:18.822585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:18.822562 2565 scope.go:117] "RemoveContainer" containerID="f591a88f734e3c51e4e29bd408e5f3464c639c5b2daa23c06df453a48dca2062" Sep 07 10:35:23.817190 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:23.817140 2565 generic.go:356] "Generic (PLEG): container finished" podID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerID="116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da" exitCode=0 Sep 07 10:35:23.817624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:23.817207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" event={"ID":"0bd2dbaa-3d01-4300-a66f-900d60e712ef","Type":"ContainerDied","Data":"116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da"} Sep 07 10:35:23.817624 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:23.817603 2565 scope.go:117] "RemoveContainer" containerID="116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da" Sep 07 10:35:23.818874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:23.818843 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:35:23.818984 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:23.818914 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:35:23.818984 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:23.818946 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:35:24.825274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:24.825240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" event={"ID":"0bd2dbaa-3d01-4300-a66f-900d60e712ef","Type":"ContainerStarted","Data":"99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825"} Sep 07 10:35:25.388247 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:25.388208 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:25.388445 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:25.388421 2565 patch_prober.go:28] interesting pod/e2e-api-b-695a13-openshift-default-984d8f756-zhng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:35:25.388500 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:25.388484 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 07 10:35:26.388511 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:26.388475 2565 patch_prober.go:28] interesting pod/e2e-api-b-695a13-openshift-default-984d8f756-zhng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:35:26.388898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:26.388533 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 07 10:35:27.387967 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:27.387925 2565 patch_prober.go:28] interesting pod/e2e-api-b-695a13-openshift-default-984d8f756-zhng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:35:27.388205 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:27.387994 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 07 10:35:28.388782 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:28.388745 2565 patch_prober.go:28] interesting pod/e2e-api-b-695a13-openshift-default-984d8f756-zhng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:35:28.389200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:28.388803 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 07 10:35:29.388636 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:29.388542 2565 patch_prober.go:28] interesting pod/e2e-api-b-695a13-openshift-default-984d8f756-zhng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:35:29.388636 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:29.388613 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 07 10:35:30.397116 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:30.397084 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:30.397963 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:30.397934 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:30.398847 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:30.398827 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:35:31.258440 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.258402 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc"] Sep 07 10:35:31.262082 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.262067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" Sep 07 10:35:31.264445 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.264428 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:35:31.264567 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.264548 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:35:31.269254 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.269232 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc"] Sep 07 10:35:31.387660 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.387627 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9083e1a-e9d9-4f78-b4c7-29421ac0a76f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc\" (UID: \"a9083e1a-e9d9-4f78-b4c7-29421ac0a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" Sep 07 10:35:31.489292 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.489258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9083e1a-e9d9-4f78-b4c7-29421ac0a76f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc\" (UID: \"a9083e1a-e9d9-4f78-b4c7-29421ac0a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" Sep 07 10:35:31.489926 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.489904 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9083e1a-e9d9-4f78-b4c7-29421ac0a76f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc\" (UID: \"a9083e1a-e9d9-4f78-b4c7-29421ac0a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" Sep 07 10:35:31.574390 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.574304 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" Sep 07 10:35:31.846418 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.846383 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc"] Sep 07 10:35:31.847759 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:35:31.847734 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9083e1a_e9d9_4f78_b4c7_29421ac0a76f.slice/crio-33be4ecfeeced5bbd5eca9ade46ebc53c0557cd929945f9a27edb4ea498c8b14 WatchSource:0}: Error finding container 33be4ecfeeced5bbd5eca9ade46ebc53c0557cd929945f9a27edb4ea498c8b14: Status 404 returned error can't find the container with id 33be4ecfeeced5bbd5eca9ade46ebc53c0557cd929945f9a27edb4ea498c8b14 Sep 07 10:35:31.876341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:31.876311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" event={"ID":"a9083e1a-e9d9-4f78-b4c7-29421ac0a76f","Type":"ContainerStarted","Data":"33be4ecfeeced5bbd5eca9ade46ebc53c0557cd929945f9a27edb4ea498c8b14"} Sep 07 10:35:32.883376 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:32.883292 2565 generic.go:356] "Generic (PLEG): container finished" podID="a9083e1a-e9d9-4f78-b4c7-29421ac0a76f" containerID="b50a481cd3631ee370808e3f830d8f82f91d7c6c786313f2ef2ae6356d377d5b" exitCode=0 Sep 07 10:35:32.883729 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:32.883377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" event={"ID":"a9083e1a-e9d9-4f78-b4c7-29421ac0a76f","Type":"ContainerDied","Data":"b50a481cd3631ee370808e3f830d8f82f91d7c6c786313f2ef2ae6356d377d5b"} Sep 07 10:35:34.024551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:34.024529 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" Sep 07 10:35:34.116339 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:34.116301 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9083e1a-e9d9-4f78-b4c7-29421ac0a76f-maas-api-service-ca\") pod \"a9083e1a-e9d9-4f78-b4c7-29421ac0a76f\" (UID: \"a9083e1a-e9d9-4f78-b4c7-29421ac0a76f\") " Sep 07 10:35:34.116656 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:34.116634 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9083e1a-e9d9-4f78-b4c7-29421ac0a76f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9083e1a-e9d9-4f78-b4c7-29421ac0a76f" (UID: "a9083e1a-e9d9-4f78-b4c7-29421ac0a76f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:35:34.217127 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:34.217058 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9083e1a-e9d9-4f78-b4c7-29421ac0a76f-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:34.895693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:34.895599 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" Sep 07 10:35:34.895693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:34.895637 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc" event={"ID":"a9083e1a-e9d9-4f78-b4c7-29421ac0a76f","Type":"ContainerDied","Data":"33be4ecfeeced5bbd5eca9ade46ebc53c0557cd929945f9a27edb4ea498c8b14"} Sep 07 10:35:34.895693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:34.895691 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33be4ecfeeced5bbd5eca9ade46ebc53c0557cd929945f9a27edb4ea498c8b14" Sep 07 10:35:47.103539 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:47.103419 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc"] Sep 07 10:35:47.106096 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:47.106074 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbea94-lrrnc"] Sep 07 10:35:48.544182 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:48.544140 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9083e1a-e9d9-4f78-b4c7-29421ac0a76f" path="/var/lib/kubelet/pods/a9083e1a-e9d9-4f78-b4c7-29421ac0a76f/volumes" Sep 07 10:35:51.917717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:51.917682 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s"] Sep 07 10:35:53.812610 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:53.812579 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95"] Sep 07 10:35:53.813064 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:53.812836 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" containerID="cri-o://1e694408a614dcfb203a003c24213502b7ce2f7df4855312e2b2e9d182f49b0a" gracePeriod=30 Sep 07 10:35:54.156088 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.156005 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt"] Sep 07 10:35:54.156484 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.156462 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9083e1a-e9d9-4f78-b4c7-29421ac0a76f" containerName="revoke-keys" Sep 07 10:35:54.156484 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.156479 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9083e1a-e9d9-4f78-b4c7-29421ac0a76f" containerName="revoke-keys" Sep 07 10:35:54.156649 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.156555 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9083e1a-e9d9-4f78-b4c7-29421ac0a76f" containerName="revoke-keys" Sep 07 10:35:54.159806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.159789 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" Sep 07 10:35:54.163784 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.163761 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:35:54.169797 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.169776 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:35:54.178665 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.178643 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt"] Sep 07 10:35:54.243117 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.243083 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e171c-c1cb-4acb-8276-9f7e30ab924f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-695a13-xn4vt\" (UID: \"bd3e171c-c1cb-4acb-8276-9f7e30ab924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" Sep 07 10:35:54.343939 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.343903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e171c-c1cb-4acb-8276-9f7e30ab924f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-695a13-xn4vt\" (UID: \"bd3e171c-c1cb-4acb-8276-9f7e30ab924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" Sep 07 10:35:54.344569 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.344551 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e171c-c1cb-4acb-8276-9f7e30ab924f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-695a13-xn4vt\" (UID: \"bd3e171c-c1cb-4acb-8276-9f7e30ab924f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" Sep 07 10:35:54.470699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.470676 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" Sep 07 10:35:54.771708 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:54.771679 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt"] Sep 07 10:35:54.773118 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:35:54.773088 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3e171c_c1cb_4acb_8276_9f7e30ab924f.slice/crio-f211b58a32e6a36e6e825605005f29be6ea7ac77d0c3645be7a1c0d75097f83d WatchSource:0}: Error finding container f211b58a32e6a36e6e825605005f29be6ea7ac77d0c3645be7a1c0d75097f83d: Status 404 returned error can't find the container with id f211b58a32e6a36e6e825605005f29be6ea7ac77d0c3645be7a1c0d75097f83d Sep 07 10:35:55.011672 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:55.011639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" event={"ID":"bd3e171c-c1cb-4acb-8276-9f7e30ab924f","Type":"ContainerStarted","Data":"f211b58a32e6a36e6e825605005f29be6ea7ac77d0c3645be7a1c0d75097f83d"} Sep 07 10:35:56.019061 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:56.019020 2565 generic.go:356] "Generic (PLEG): container finished" podID="bd3e171c-c1cb-4acb-8276-9f7e30ab924f" containerID="0b16c20b80e77b89eb64573ebada83dde6b27ff715b9df33a77a3a765af9521c" exitCode=0 Sep 07 10:35:56.019502 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:56.019104 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" event={"ID":"bd3e171c-c1cb-4acb-8276-9f7e30ab924f","Type":"ContainerDied","Data":"0b16c20b80e77b89eb64573ebada83dde6b27ff715b9df33a77a3a765af9521c"} Sep 07 10:35:57.167329 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:57.167304 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" Sep 07 10:35:57.276415 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:57.276329 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e171c-c1cb-4acb-8276-9f7e30ab924f-maas-api-service-ca\") pod \"bd3e171c-c1cb-4acb-8276-9f7e30ab924f\" (UID: \"bd3e171c-c1cb-4acb-8276-9f7e30ab924f\") " Sep 07 10:35:57.276755 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:57.276722 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3e171c-c1cb-4acb-8276-9f7e30ab924f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd3e171c-c1cb-4acb-8276-9f7e30ab924f" (UID: "bd3e171c-c1cb-4acb-8276-9f7e30ab924f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:35:57.377424 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:57.377392 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3e171c-c1cb-4acb-8276-9f7e30ab924f-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:58.032022 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:58.031987 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" event={"ID":"bd3e171c-c1cb-4acb-8276-9f7e30ab924f","Type":"ContainerDied","Data":"f211b58a32e6a36e6e825605005f29be6ea7ac77d0c3645be7a1c0d75097f83d"} Sep 07 10:35:58.032022 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:58.032015 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt" Sep 07 10:35:58.032262 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:58.032046 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f211b58a32e6a36e6e825605005f29be6ea7ac77d0c3645be7a1c0d75097f83d" Sep 07 10:35:59.039961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.039929 2565 generic.go:356] "Generic (PLEG): container finished" podID="f7237ca7-155b-4512-9708-379a343ff40b" containerID="1e694408a614dcfb203a003c24213502b7ce2f7df4855312e2b2e9d182f49b0a" exitCode=0 Sep 07 10:35:59.040392 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.040005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" event={"ID":"f7237ca7-155b-4512-9708-379a343ff40b","Type":"ContainerDied","Data":"1e694408a614dcfb203a003c24213502b7ce2f7df4855312e2b2e9d182f49b0a"} Sep 07 10:35:59.040392 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.040066 2565 scope.go:117] "RemoveContainer" containerID="3af6416c4e4179330aad034c5e45593e3eb72b547deee94dd6dd7f5a6d39ffc8" Sep 07 10:35:59.090671 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.090645 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:35:59.196284 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196251 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7237ca7-155b-4512-9708-379a343ff40b-istiod-ca-cert\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196475 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196312 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-data\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196475 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196350 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-credential-socket\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196475 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196377 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-certs\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196475 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196417 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-envoy\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196492 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb6jv\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-kube-api-access-gb6jv\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196534 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-istio-token\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196566 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7237ca7-155b-4512-9708-379a343ff40b-istio-podinfo\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196592 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-socket\") pod \"f7237ca7-155b-4512-9708-379a343ff40b\" (UID: \"f7237ca7-155b-4512-9708-379a343ff40b\") " Sep 07 10:35:59.196703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196664 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:35:59.196935 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196748 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:35:59.196935 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196782 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7237ca7-155b-4512-9708-379a343ff40b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:35:59.197033 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196936 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:35:59.197033 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.196992 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:35:59.197149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.197055 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:59.197149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.197069 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:59.197149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.197083 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7237ca7-155b-4512-9708-379a343ff40b-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:59.197149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.197092 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:59.198914 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.198889 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7237ca7-155b-4512-9708-379a343ff40b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:35:59.199033 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.198944 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:35:59.199267 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.199248 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:35:59.199325 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.199279 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-kube-api-access-gb6jv" (OuterVolumeSpecName: "kube-api-access-gb6jv") pod "f7237ca7-155b-4512-9708-379a343ff40b" (UID: "f7237ca7-155b-4512-9708-379a343ff40b"). InnerVolumeSpecName "kube-api-access-gb6jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:35:59.298596 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.298569 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:59.298596 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.298592 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb6jv\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-kube-api-access-gb6jv\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:59.298763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.298602 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7237ca7-155b-4512-9708-379a343ff40b-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:59.298763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.298611 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7237ca7-155b-4512-9708-379a343ff40b-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:35:59.298763 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:35:59.298619 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7237ca7-155b-4512-9708-379a343ff40b-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:00.047125 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:00.047097 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" Sep 07 10:36:00.047574 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:00.047097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95" event={"ID":"f7237ca7-155b-4512-9708-379a343ff40b","Type":"ContainerDied","Data":"4e9f36855aed41e143adf418ec988e044ac8824802974cbe1bd0c621f6756f6a"} Sep 07 10:36:00.047574 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:00.047230 2565 scope.go:117] "RemoveContainer" containerID="1e694408a614dcfb203a003c24213502b7ce2f7df4855312e2b2e9d182f49b0a" Sep 07 10:36:00.069515 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:00.069488 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95"] Sep 07 10:36:00.074052 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:00.074022 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fbea94-openshift-default-ccfcf964b-wwb95"] Sep 07 10:36:00.550687 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:00.550653 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7237ca7-155b-4512-9708-379a343ff40b" path="/var/lib/kubelet/pods/f7237ca7-155b-4512-9708-379a343ff40b/volumes" Sep 07 10:36:06.938360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:06.938314 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" podUID="62f1a321-96a5-4205-af95-c7d44e1cc110" containerName="main" containerID="cri-o://614ee306292599cf773d4af2fda0590bc5a9893f2a7ecca251861be1f088d975" gracePeriod=45 Sep 07 10:36:07.092468 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.092431 2565 generic.go:356] "Generic (PLEG): container finished" podID="62f1a321-96a5-4205-af95-c7d44e1cc110" containerID="614ee306292599cf773d4af2fda0590bc5a9893f2a7ecca251861be1f088d975" exitCode=0 Sep 07 10:36:07.092635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.092509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" event={"ID":"62f1a321-96a5-4205-af95-c7d44e1cc110","Type":"ContainerDied","Data":"614ee306292599cf773d4af2fda0590bc5a9893f2a7ecca251861be1f088d975"} Sep 07 10:36:07.204666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.204601 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:36:07.279744 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.279706 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-model-cache\") pod \"62f1a321-96a5-4205-af95-c7d44e1cc110\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " Sep 07 10:36:07.279744 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.279751 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vfm6\" (UniqueName: \"kubernetes.io/projected/62f1a321-96a5-4205-af95-c7d44e1cc110-kube-api-access-6vfm6\") pod \"62f1a321-96a5-4205-af95-c7d44e1cc110\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " Sep 07 10:36:07.279996 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.279794 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-home\") pod \"62f1a321-96a5-4205-af95-c7d44e1cc110\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " Sep 07 10:36:07.279996 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.279829 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f1a321-96a5-4205-af95-c7d44e1cc110-tls-certs\") pod \"62f1a321-96a5-4205-af95-c7d44e1cc110\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " Sep 07 10:36:07.279996 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.279912 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-tmp-dir\") pod \"62f1a321-96a5-4205-af95-c7d44e1cc110\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " Sep 07 10:36:07.279996 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.279982 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-dshm\") pod \"62f1a321-96a5-4205-af95-c7d44e1cc110\" (UID: \"62f1a321-96a5-4205-af95-c7d44e1cc110\") " Sep 07 10:36:07.280262 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.279989 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-model-cache" (OuterVolumeSpecName: "model-cache") pod "62f1a321-96a5-4205-af95-c7d44e1cc110" (UID: "62f1a321-96a5-4205-af95-c7d44e1cc110"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:07.280262 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.280100 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-home" (OuterVolumeSpecName: "home") pod "62f1a321-96a5-4205-af95-c7d44e1cc110" (UID: "62f1a321-96a5-4205-af95-c7d44e1cc110"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:07.280262 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.280237 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62f1a321-96a5-4205-af95-c7d44e1cc110" (UID: "62f1a321-96a5-4205-af95-c7d44e1cc110"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:07.280412 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.280395 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:07.280466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.280418 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:07.280466 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.280431 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:07.282249 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.282213 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-dshm" (OuterVolumeSpecName: "dshm") pod "62f1a321-96a5-4205-af95-c7d44e1cc110" (UID: "62f1a321-96a5-4205-af95-c7d44e1cc110"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:07.282371 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.282294 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f1a321-96a5-4205-af95-c7d44e1cc110-kube-api-access-6vfm6" (OuterVolumeSpecName: "kube-api-access-6vfm6") pod "62f1a321-96a5-4205-af95-c7d44e1cc110" (UID: "62f1a321-96a5-4205-af95-c7d44e1cc110"). InnerVolumeSpecName "kube-api-access-6vfm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:36:07.282425 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.282401 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f1a321-96a5-4205-af95-c7d44e1cc110-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62f1a321-96a5-4205-af95-c7d44e1cc110" (UID: "62f1a321-96a5-4205-af95-c7d44e1cc110"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:36:07.381719 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.381678 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vfm6\" (UniqueName: \"kubernetes.io/projected/62f1a321-96a5-4205-af95-c7d44e1cc110-kube-api-access-6vfm6\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:07.381719 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.381711 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f1a321-96a5-4205-af95-c7d44e1cc110-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:07.381719 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:07.381724 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f1a321-96a5-4205-af95-c7d44e1cc110-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:08.099868 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:08.099831 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" Sep 07 10:36:08.100363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:08.099877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s" event={"ID":"62f1a321-96a5-4205-af95-c7d44e1cc110","Type":"ContainerDied","Data":"86d01999324f423535dbc61134b8ecebac0a16109b9301c6379f1b4650a972a8"} Sep 07 10:36:08.100363 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:08.099935 2565 scope.go:117] "RemoveContainer" containerID="614ee306292599cf773d4af2fda0590bc5a9893f2a7ecca251861be1f088d975" Sep 07 10:36:08.134500 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:08.134472 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s"] Sep 07 10:36:08.151632 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:08.151607 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-93e41e/sub-test-model-93e41e-kserve-7db5b68f4f-9cx7s"] Sep 07 10:36:08.544326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:08.544292 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f1a321-96a5-4205-af95-c7d44e1cc110" path="/var/lib/kubelet/pods/62f1a321-96a5-4205-af95-c7d44e1cc110/volumes" Sep 07 10:36:10.331084 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:10.331048 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt"] Sep 07 10:36:10.338734 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:10.338709 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-695a13-xn4vt"] Sep 07 10:36:10.544420 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:10.544380 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3e171c-c1cb-4acb-8276-9f7e30ab924f" path="/var/lib/kubelet/pods/bd3e171c-c1cb-4acb-8276-9f7e30ab924f/volumes" Sep 07 10:36:18.553925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.553895 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2"] Sep 07 10:36:18.554366 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.554112 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" containerID="cri-o://99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825" gracePeriod=30 Sep 07 10:36:18.633225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.633193 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:36:18.636119 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.636096 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:36:18.927013 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.926977 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj"] Sep 07 10:36:18.927487 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927473 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" Sep 07 10:36:18.927542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927490 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" Sep 07 10:36:18.927542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927499 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3e171c-c1cb-4acb-8276-9f7e30ab924f" containerName="revoke-keys" Sep 07 10:36:18.927542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927508 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3e171c-c1cb-4acb-8276-9f7e30ab924f" containerName="revoke-keys" Sep 07 10:36:18.927542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927523 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f1a321-96a5-4205-af95-c7d44e1cc110" containerName="main" Sep 07 10:36:18.927542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927529 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f1a321-96a5-4205-af95-c7d44e1cc110" containerName="main" Sep 07 10:36:18.927542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927537 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" Sep 07 10:36:18.927542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927542 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" Sep 07 10:36:18.927759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927616 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" Sep 07 10:36:18.927759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927623 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7237ca7-155b-4512-9708-379a343ff40b" containerName="istio-proxy" Sep 07 10:36:18.927759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927632 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3e171c-c1cb-4acb-8276-9f7e30ab924f" containerName="revoke-keys" Sep 07 10:36:18.927759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.927641 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f1a321-96a5-4205-af95-c7d44e1cc110" containerName="main" Sep 07 10:36:18.931936 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.931919 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:18.939882 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:36:18.939847 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 07 10:36:18.939882 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:36:18.939851 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="e5c53b22-ced0-4a60-9f8e-1a94a6c4111b" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:18.940035 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.939928 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:36:18.955745 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:18.955715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj"] Sep 07 10:36:19.012330 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:19.012290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj\" (UID: \"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:19.112947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:19.112909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj\" (UID: \"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:20.114065 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:36:20.114031 2565 configmap.go:193] Couldn't get configMap odh-ai-gateway-infra/openshift-service-ca.crt: failed to sync configmap cache: timed out waiting for the condition Sep 07 10:36:20.114508 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:36:20.114128 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca podName:e5c53b22-ced0-4a60-9f8e-1a94a6c4111b nodeName:}" failed. No retries permitted until 2026-09-07 10:36:20.614106784 +0000 UTC m=+1502.600387103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-service-ca" (UniqueName: "kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca") pod "maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" (UID: "e5c53b22-ced0-4a60-9f8e-1a94a6c4111b") : failed to sync configmap cache: timed out waiting for the condition Sep 07 10:36:20.412452 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:20.412374 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:36:20.629154 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:20.629083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj\" (UID: \"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:20.629885 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:20.629859 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj\" (UID: \"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:20.742608 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:20.742577 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:21.018961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:21.018934 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj"] Sep 07 10:36:21.020630 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:36:21.020596 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c53b22_ced0_4a60_9f8e_1a94a6c4111b.slice/crio-69ba0c88f31710075b9555d8e27f53e915b10b709c0f815d56749cf95405b65c WatchSource:0}: Error finding container 69ba0c88f31710075b9555d8e27f53e915b10b709c0f815d56749cf95405b65c: Status 404 returned error can't find the container with id 69ba0c88f31710075b9555d8e27f53e915b10b709c0f815d56749cf95405b65c Sep 07 10:36:21.175808 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:21.175772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" event={"ID":"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b","Type":"ContainerStarted","Data":"69ba0c88f31710075b9555d8e27f53e915b10b709c0f815d56749cf95405b65c"} Sep 07 10:36:22.182897 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:22.182806 2565 generic.go:356] "Generic (PLEG): container finished" podID="e5c53b22-ced0-4a60-9f8e-1a94a6c4111b" containerID="5b2911c6e18135db168226022bd731120913098d083f613ca0ade842f66c2fd1" exitCode=0 Sep 07 10:36:22.183316 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:22.182891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" event={"ID":"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b","Type":"ContainerDied","Data":"5b2911c6e18135db168226022bd731120913098d083f613ca0ade842f66c2fd1"} Sep 07 10:36:23.354931 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.354905 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:23.456054 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.455956 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca\") pod \"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b\" (UID: \"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b\") " Sep 07 10:36:23.456367 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.456345 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5c53b22-ced0-4a60-9f8e-1a94a6c4111b" (UID: "e5c53b22-ced0-4a60-9f8e-1a94a6c4111b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:36:23.556793 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.556756 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.822479 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.822447 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:36:23.859898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.859865 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-envoy\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860105 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.859921 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggs2f\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-kube-api-access-ggs2f\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860105 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.859960 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-podinfo\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860105 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.860003 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-certs\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860105 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.860029 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-token\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860105 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.860068 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-socket\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860406 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.860112 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istiod-ca-cert\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860406 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.860140 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-credential-socket\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860406 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.860197 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-data\") pod \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\" (UID: \"0bd2dbaa-3d01-4300-a66f-900d60e712ef\") " Sep 07 10:36:23.860406 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.860386 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:23.860664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.860638 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:23.862405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.862213 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.862405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.862239 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.862799 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.862651 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:23.863111 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.862980 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-kube-api-access-ggs2f" (OuterVolumeSpecName: "kube-api-access-ggs2f") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "kube-api-access-ggs2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:36:23.863111 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.863008 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:36:23.863366 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.863235 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:23.863366 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.863325 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:23.867214 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.864223 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:36:23.870216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.868176 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0bd2dbaa-3d01-4300-a66f-900d60e712ef" (UID: "0bd2dbaa-3d01-4300-a66f-900d60e712ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:36:23.962907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.962845 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.962907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.962901 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.962907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.962911 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.962907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.962920 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.962907 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.962930 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggs2f\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-kube-api-access-ggs2f\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.963283 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.962938 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:23.963283 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:23.962947 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd2dbaa-3d01-4300-a66f-900d60e712ef-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:24.200861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.200774 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" Sep 07 10:36:24.200861 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.200811 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj" event={"ID":"e5c53b22-ced0-4a60-9f8e-1a94a6c4111b","Type":"ContainerDied","Data":"69ba0c88f31710075b9555d8e27f53e915b10b709c0f815d56749cf95405b65c"} Sep 07 10:36:24.201080 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.200887 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ba0c88f31710075b9555d8e27f53e915b10b709c0f815d56749cf95405b65c" Sep 07 10:36:24.202662 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.202637 2565 generic.go:356] "Generic (PLEG): container finished" podID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerID="99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825" exitCode=0 Sep 07 10:36:24.202775 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.202679 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" event={"ID":"0bd2dbaa-3d01-4300-a66f-900d60e712ef","Type":"ContainerDied","Data":"99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825"} Sep 07 10:36:24.202775 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.202711 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" Sep 07 10:36:24.202775 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.202731 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2" event={"ID":"0bd2dbaa-3d01-4300-a66f-900d60e712ef","Type":"ContainerDied","Data":"ee63d5f5660a8ab2f5af4f49ba2c431642cb77a5f1e063161e12865b01a24235"} Sep 07 10:36:24.202775 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.202754 2565 scope.go:117] "RemoveContainer" containerID="99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825" Sep 07 10:36:24.213502 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.213477 2565 scope.go:117] "RemoveContainer" containerID="116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da" Sep 07 10:36:24.223065 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.223043 2565 scope.go:117] "RemoveContainer" containerID="99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825" Sep 07 10:36:24.223466 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:36:24.223445 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825\": container with ID starting with 99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825 not found: ID does not exist" containerID="99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825" Sep 07 10:36:24.223520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.223477 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825"} err="failed to get container status \"99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825\": rpc error: code = NotFound desc = could not find container \"99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825\": container with ID starting with 99393aa1ae14c16bb291b7d43827248b44707ecf2a20e687ad00ec81dfd0a825 not found: ID does not exist" Sep 07 10:36:24.223520 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.223497 2565 scope.go:117] "RemoveContainer" containerID="116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da" Sep 07 10:36:24.223784 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:36:24.223766 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da\": container with ID starting with 116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da not found: ID does not exist" containerID="116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da" Sep 07 10:36:24.223832 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.223790 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da"} err="failed to get container status \"116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da\": rpc error: code = NotFound desc = could not find container \"116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da\": container with ID starting with 116ea818e62c63f6771e05326fd97c888389506e74679de8da95f771751455da not found: ID does not exist" Sep 07 10:36:24.252949 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.252915 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2"] Sep 07 10:36:24.260975 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.260933 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-695a13-openshift-default-984d8f756-zhng2"] Sep 07 10:36:24.544850 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:24.544819 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" path="/var/lib/kubelet/pods/0bd2dbaa-3d01-4300-a66f-900d60e712ef/volumes" Sep 07 10:36:29.919182 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:29.919137 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq"] Sep 07 10:36:29.926351 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:29.926324 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bdcc7-29812950-qb9nq"] Sep 07 10:36:30.339427 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:30.339395 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj"] Sep 07 10:36:30.344713 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:30.344688 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bdcc7-rz9wj"] Sep 07 10:36:30.544816 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:30.544784 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6453be-7d00-45d5-bac2-37aa0c654c37" path="/var/lib/kubelet/pods/2e6453be-7d00-45d5-bac2-37aa0c654c37/volumes" Sep 07 10:36:30.545105 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:30.545093 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c53b22-ced0-4a60-9f8e-1a94a6c4111b" path="/var/lib/kubelet/pods/e5c53b22-ced0-4a60-9f8e-1a94a6c4111b/volumes" Sep 07 10:36:38.641709 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.641676 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql"] Sep 07 10:36:38.642248 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.641904 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" containerID="cri-o://44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3" gracePeriod=30 Sep 07 10:36:38.970201 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.970102 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv"] Sep 07 10:36:38.970570 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.970556 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c53b22-ced0-4a60-9f8e-1a94a6c4111b" containerName="revoke-keys" Sep 07 10:36:38.970635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.970573 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c53b22-ced0-4a60-9f8e-1a94a6c4111b" containerName="revoke-keys" Sep 07 10:36:38.970635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.970584 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" Sep 07 10:36:38.970635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.970591 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" Sep 07 10:36:38.970733 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.970679 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" Sep 07 10:36:38.970733 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.970690 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c53b22-ced0-4a60-9f8e-1a94a6c4111b" containerName="revoke-keys" Sep 07 10:36:38.970733 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.970700 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" Sep 07 10:36:38.975555 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.975527 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" Sep 07 10:36:38.978316 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.978294 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:36:38.978670 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.978652 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:36:38.990215 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:38.990186 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv"] Sep 07 10:36:39.112850 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:39.112805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28280993-bdf1-43ab-b72a-daf35e52d709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv\" (UID: \"28280993-bdf1-43ab-b72a-daf35e52d709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" Sep 07 10:36:39.213670 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:39.213621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28280993-bdf1-43ab-b72a-daf35e52d709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv\" (UID: \"28280993-bdf1-43ab-b72a-daf35e52d709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" Sep 07 10:36:39.214353 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:39.214329 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28280993-bdf1-43ab-b72a-daf35e52d709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv\" (UID: \"28280993-bdf1-43ab-b72a-daf35e52d709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" Sep 07 10:36:39.288108 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:39.288074 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" Sep 07 10:36:39.629769 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:39.629682 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv"] Sep 07 10:36:39.640075 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:36:39.640047 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28280993_bdf1_43ab_b72a_daf35e52d709.slice/crio-a87d7b5c816020395b3ef5e92f271690bfa68900cc55428c303d0866ba3e13be WatchSource:0}: Error finding container a87d7b5c816020395b3ef5e92f271690bfa68900cc55428c303d0866ba3e13be: Status 404 returned error can't find the container with id a87d7b5c816020395b3ef5e92f271690bfa68900cc55428c303d0866ba3e13be Sep 07 10:36:40.297633 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:40.297591 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" event={"ID":"28280993-bdf1-43ab-b72a-daf35e52d709","Type":"ContainerStarted","Data":"e2c5f1539bf63466c30bb571f33b9470bb55cc54be4af7454eb0c0905f9b3b48"} Sep 07 10:36:40.298054 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:40.297666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" event={"ID":"28280993-bdf1-43ab-b72a-daf35e52d709","Type":"ContainerStarted","Data":"a87d7b5c816020395b3ef5e92f271690bfa68900cc55428c303d0866ba3e13be"} Sep 07 10:36:40.330597 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:40.330543 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" podStartSLOduration=2.33052571 podStartE2EDuration="2.33052571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:36:40.329013016 +0000 UTC m=+1522.315293349" watchObservedRunningTime="2026-09-07 10:36:40.33052571 +0000 UTC m=+1522.316806105" Sep 07 10:36:41.304800 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:41.304761 2565 generic.go:356] "Generic (PLEG): container finished" podID="28280993-bdf1-43ab-b72a-daf35e52d709" containerID="e2c5f1539bf63466c30bb571f33b9470bb55cc54be4af7454eb0c0905f9b3b48" exitCode=0 Sep 07 10:36:41.305235 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:41.304819 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" event={"ID":"28280993-bdf1-43ab-b72a-daf35e52d709","Type":"ContainerDied","Data":"e2c5f1539bf63466c30bb571f33b9470bb55cc54be4af7454eb0c0905f9b3b48"} Sep 07 10:36:42.449332 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:42.449304 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" Sep 07 10:36:42.547181 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:42.547141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28280993-bdf1-43ab-b72a-daf35e52d709-maas-api-service-ca\") pod \"28280993-bdf1-43ab-b72a-daf35e52d709\" (UID: \"28280993-bdf1-43ab-b72a-daf35e52d709\") " Sep 07 10:36:42.547467 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:42.547445 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28280993-bdf1-43ab-b72a-daf35e52d709-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28280993-bdf1-43ab-b72a-daf35e52d709" (UID: "28280993-bdf1-43ab-b72a-daf35e52d709"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:36:42.648019 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:42.647935 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28280993-bdf1-43ab-b72a-daf35e52d709-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:43.319586 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:43.319559 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" Sep 07 10:36:43.319757 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:43.319599 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv" event={"ID":"28280993-bdf1-43ab-b72a-daf35e52d709","Type":"ContainerDied","Data":"a87d7b5c816020395b3ef5e92f271690bfa68900cc55428c303d0866ba3e13be"} Sep 07 10:36:43.319757 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:43.319646 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87d7b5c816020395b3ef5e92f271690bfa68900cc55428c303d0866ba3e13be" Sep 07 10:36:43.906329 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:43.906306 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:36:44.062572 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-credential-socket\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.062756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062598 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-data\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.062756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062620 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-token\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.062756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062683 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b9ddb-1092-40f9-a889-5e4107b4d84e-istiod-ca-cert\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.062756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-envoy\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.062756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062716 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-socket\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.063025 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062801 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsj5d\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-kube-api-access-hsj5d\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.063025 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062839 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-certs\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.063025 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062873 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-podinfo\") pod \"661b9ddb-1092-40f9-a889-5e4107b4d84e\" (UID: \"661b9ddb-1092-40f9-a889-5e4107b4d84e\") " Sep 07 10:36:44.063025 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.062871 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:44.063291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.063039 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:44.063291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.063125 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:44.063291 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.063133 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661b9ddb-1092-40f9-a889-5e4107b4d84e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:36:44.063454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.063301 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-data" (OuterVolumeSpecName: "istio-data") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:44.063454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.063316 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.063454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.063359 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661b9ddb-1092-40f9-a889-5e4107b4d84e-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.063454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.063376 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.063454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.063388 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.065129 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.065102 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:36:44.065271 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.065153 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-kube-api-access-hsj5d" (OuterVolumeSpecName: "kube-api-access-hsj5d") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "kube-api-access-hsj5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:36:44.065499 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.065481 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:36:44.065548 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.065530 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-token" (OuterVolumeSpecName: "istio-token") pod "661b9ddb-1092-40f9-a889-5e4107b4d84e" (UID: "661b9ddb-1092-40f9-a889-5e4107b4d84e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:36:44.163915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.163876 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.163915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.163909 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.163915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.163917 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.163915 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.163926 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsj5d\" (UniqueName: \"kubernetes.io/projected/661b9ddb-1092-40f9-a889-5e4107b4d84e-kube-api-access-hsj5d\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.164225 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.163936 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661b9ddb-1092-40f9-a889-5e4107b4d84e-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:36:44.328736 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.328639 2565 generic.go:356] "Generic (PLEG): container finished" podID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerID="44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3" exitCode=0 Sep 07 10:36:44.328736 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.328701 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" Sep 07 10:36:44.328736 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.328720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" event={"ID":"661b9ddb-1092-40f9-a889-5e4107b4d84e","Type":"ContainerDied","Data":"44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3"} Sep 07 10:36:44.328990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.328779 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql" event={"ID":"661b9ddb-1092-40f9-a889-5e4107b4d84e","Type":"ContainerDied","Data":"2138e918c5189d79b1de3cf1fc4a83a6e3c35ea68fc278979f99704b39bafa35"} Sep 07 10:36:44.328990 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.328812 2565 scope.go:117] "RemoveContainer" containerID="44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3" Sep 07 10:36:44.339479 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.339460 2565 scope.go:117] "RemoveContainer" containerID="94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc" Sep 07 10:36:44.348615 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.348595 2565 scope.go:117] "RemoveContainer" containerID="44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3" Sep 07 10:36:44.348958 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:36:44.348937 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3\": container with ID starting with 44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3 not found: ID does not exist" containerID="44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3" Sep 07 10:36:44.349023 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.348971 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3"} err="failed to get container status \"44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3\": rpc error: code = NotFound desc = could not find container \"44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3\": container with ID starting with 44972a8a68f9f0ce300c2e32666606d4615a2ff2c0867c36b9a775d6c6dd81e3 not found: ID does not exist" Sep 07 10:36:44.349023 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.348990 2565 scope.go:117] "RemoveContainer" containerID="94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc" Sep 07 10:36:44.349436 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:36:44.349416 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc\": container with ID starting with 94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc not found: ID does not exist" containerID="94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc" Sep 07 10:36:44.349515 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.349441 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc"} err="failed to get container status \"94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc\": rpc error: code = NotFound desc = could not find container \"94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc\": container with ID starting with 94ef753e3184f56ad3aff317e74941b58c24e5ada8b39fabd279b8ede1dceacc not found: ID does not exist" Sep 07 10:36:44.384911 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.384871 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql"] Sep 07 10:36:44.387074 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.387050 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7bdcc7-openshift-default-66bb59b8f4-npcql"] Sep 07 10:36:44.544341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:44.544308 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" path="/var/lib/kubelet/pods/661b9ddb-1092-40f9-a889-5e4107b4d84e/volumes" Sep 07 10:36:50.530133 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:50.530093 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv"] Sep 07 10:36:50.535570 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:50.535543 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-319d4c-rzqcv"] Sep 07 10:36:50.543433 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:50.543406 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28280993-bdf1-43ab-b72a-daf35e52d709" path="/var/lib/kubelet/pods/28280993-bdf1-43ab-b72a-daf35e52d709/volumes" Sep 07 10:36:58.923635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:58.923601 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv"] Sep 07 10:36:58.924385 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:36:58.923818 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" containerID="cri-o://8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7" gracePeriod=30 Sep 07 10:37:02.468626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:02.468586 2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 10:37:02.469024 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:02.468654 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:37:04.178082 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.178055 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:37:04.255972 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.255938 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-certs\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.255972 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.255976 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-podinfo\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.256281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256010 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1e1f-6459-4f77-8d03-d448a6e90964-istiod-ca-cert\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.256281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256057 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-credential-socket\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.256281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256110 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp7vh\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-kube-api-access-gp7vh\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.256281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-envoy\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.256281 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256276 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-token\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.256547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256306 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-data\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.256547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256356 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:37:04.256547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256364 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-socket\") pod \"77bd1e1f-6459-4f77-8d03-d448a6e90964\" (UID: \"77bd1e1f-6459-4f77-8d03-d448a6e90964\") " Sep 07 10:37:04.256547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256412 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:37:04.256547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256508 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bd1e1f-6459-4f77-8d03-d448a6e90964-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:37:04.256774 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256662 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-data" (OuterVolumeSpecName: "istio-data") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:37:04.256774 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256711 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:37:04.256988 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.256895 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.257079 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.257064 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.257123 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.257084 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.257123 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.257098 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.257123 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.257112 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bd1e1f-6459-4f77-8d03-d448a6e90964-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.258445 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.258422 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-kube-api-access-gp7vh" (OuterVolumeSpecName: "kube-api-access-gp7vh") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "kube-api-access-gp7vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:37:04.258542 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.258470 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-token" (OuterVolumeSpecName: "istio-token") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:37:04.258877 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.258859 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:37:04.258952 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.258860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77bd1e1f-6459-4f77-8d03-d448a6e90964" (UID: "77bd1e1f-6459-4f77-8d03-d448a6e90964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:37:04.358581 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.358538 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7vh\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-kube-api-access-gp7vh\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.358581 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.358570 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.358581 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.358579 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.358581 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.358587 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bd1e1f-6459-4f77-8d03-d448a6e90964-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:37:04.447314 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.447276 2565 generic.go:356] "Generic (PLEG): container finished" podID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerID="8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7" exitCode=0 Sep 07 10:37:04.447484 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.447333 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" Sep 07 10:37:04.447484 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.447363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" event={"ID":"77bd1e1f-6459-4f77-8d03-d448a6e90964","Type":"ContainerDied","Data":"8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7"} Sep 07 10:37:04.447484 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.447418 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv" event={"ID":"77bd1e1f-6459-4f77-8d03-d448a6e90964","Type":"ContainerDied","Data":"a300bde6cfa90d898fa4c905d5e11bb497bb79c42d537b68691463c77adf3768"} Sep 07 10:37:04.447484 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.447441 2565 scope.go:117] "RemoveContainer" containerID="8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7" Sep 07 10:37:04.465357 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.465334 2565 scope.go:117] "RemoveContainer" containerID="2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521" Sep 07 10:37:04.474497 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.474475 2565 scope.go:117] "RemoveContainer" containerID="8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7" Sep 07 10:37:04.474814 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:37:04.474793 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7\": container with ID starting with 8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7 not found: ID does not exist" containerID="8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7" Sep 07 10:37:04.474872 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.474825 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7"} err="failed to get container status \"8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7\": rpc error: code = NotFound desc = could not find container \"8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7\": container with ID starting with 8df663d9a361101c6562485dbcfe9304a2d31702c837f1790e4f20d2dc758ec7 not found: ID does not exist" Sep 07 10:37:04.474872 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.474845 2565 scope.go:117] "RemoveContainer" containerID="2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521" Sep 07 10:37:04.475119 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:37:04.475102 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521\": container with ID starting with 2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521 not found: ID does not exist" containerID="2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521" Sep 07 10:37:04.475213 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.475125 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521"} err="failed to get container status \"2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521\": rpc error: code = NotFound desc = could not find container \"2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521\": container with ID starting with 2f931482aadc2afbc8bd607cc8a111d9b3d57a53651601d66c808b9d86738521 not found: ID does not exist" Sep 07 10:37:04.483192 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.483145 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv"] Sep 07 10:37:04.491112 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.491075 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-319d4c-openshift-default-6854578bc6-p24cv"] Sep 07 10:37:04.544448 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:04.544361 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" path="/var/lib/kubelet/pods/77bd1e1f-6459-4f77-8d03-d448a6e90964/volumes" Sep 07 10:37:18.889068 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:18.889036 2565 scope.go:117] "RemoveContainer" containerID="889553c09e503167ea0407fae42f23f90d84060317c405128f2c57b3f0bcd0f0" Sep 07 10:37:18.897959 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:18.897935 2565 scope.go:117] "RemoveContainer" containerID="f48af4d78d75270d968666379bba4a63c4c8629a09054917122a39ee0e778607" Sep 07 10:37:26.904343 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904309 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn"] Sep 07 10:37:26.904766 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904693 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" Sep 07 10:37:26.904766 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904704 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" Sep 07 10:37:26.904766 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904712 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" Sep 07 10:37:26.904766 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904718 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd2dbaa-3d01-4300-a66f-900d60e712ef" containerName="istio-proxy" Sep 07 10:37:26.904766 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904738 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" Sep 07 10:37:26.904766 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904744 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" Sep 07 10:37:26.904766 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904750 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28280993-bdf1-43ab-b72a-daf35e52d709" containerName="revoke-keys" Sep 07 10:37:26.904766 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904755 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="28280993-bdf1-43ab-b72a-daf35e52d709" containerName="revoke-keys" Sep 07 10:37:26.905034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904820 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" Sep 07 10:37:26.905034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904828 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" Sep 07 10:37:26.905034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904833 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" Sep 07 10:37:26.905034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.904842 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="28280993-bdf1-43ab-b72a-daf35e52d709" containerName="revoke-keys" Sep 07 10:37:26.909557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.909541 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.915639 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.915614 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-92aa2e-openshift-default-dockercfg-w4dts\"" Sep 07 10:37:26.945961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.945928 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn"] Sep 07 10:37:26.984717 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.984681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istiod-ca-cert\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.984898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.984777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-podinfo\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.984898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.984837 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-socket\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.984898 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.984858 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-envoy\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.985009 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.984912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcs6\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-kube-api-access-hqcs6\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.985009 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.984944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-data\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.985009 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.984997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-credential-socket\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.985106 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.985020 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-token\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:26.985106 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:26.985052 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-certs\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.085945 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.085912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-credential-socket\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.085945 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.085950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-token\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.085980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-certs\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istiod-ca-cert\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-podinfo\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-socket\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086185 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-envoy\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086222 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcs6\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-kube-api-access-hqcs6\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-data\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-credential-socket\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086419 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-certs\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-socket\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086762 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-data\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.086799 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.086782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istiod-ca-cert\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.088541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.088518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-envoy\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.088759 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.088740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-podinfo\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.106032 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.106009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-token\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.106135 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.106082 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcs6\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-kube-api-access-hqcs6\") pod \"e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.219397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.219309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:27.514961 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.514937 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn"] Sep 07 10:37:27.524828 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:37:27.524606 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a47ee8_6be1_4b0e_93f5_fa4fda879d5e.slice/crio-05f23ee1862799c82018b62d0a8af9ba45ad3bbaf374d7e2b0f34bfbb5e9fdec WatchSource:0}: Error finding container 05f23ee1862799c82018b62d0a8af9ba45ad3bbaf374d7e2b0f34bfbb5e9fdec: Status 404 returned error can't find the container with id 05f23ee1862799c82018b62d0a8af9ba45ad3bbaf374d7e2b0f34bfbb5e9fdec Sep 07 10:37:27.526424 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.526391 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:27.526492 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.526457 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:27.526492 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.526486 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:27.583534 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:27.583498 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" event={"ID":"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e","Type":"ContainerStarted","Data":"05f23ee1862799c82018b62d0a8af9ba45ad3bbaf374d7e2b0f34bfbb5e9fdec"} Sep 07 10:37:28.590030 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:28.589998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" event={"ID":"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e","Type":"ContainerStarted","Data":"2715620e687748858d2f978088907071e3d563412584746bbc6d8b4b028a0d16"} Sep 07 10:37:28.659927 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:28.659876 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" podStartSLOduration=2.659861228 podStartE2EDuration="2.659861228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:37:28.6595078 +0000 UTC m=+1570.645788145" watchObservedRunningTime="2026-09-07 10:37:28.659861228 +0000 UTC m=+1570.646141562" Sep 07 10:37:29.219918 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:29.219880 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:29.224887 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:29.224860 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:29.595657 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:29.595623 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:29.596675 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:29.596656 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:45.637379 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:45.637335 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:37:45.637787 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:45.637415 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:37:46.699002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:46.698910 2565 generic.go:356] "Generic (PLEG): container finished" podID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerID="2715620e687748858d2f978088907071e3d563412584746bbc6d8b4b028a0d16" exitCode=0 Sep 07 10:37:46.699002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:46.698949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" event={"ID":"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e","Type":"ContainerDied","Data":"2715620e687748858d2f978088907071e3d563412584746bbc6d8b4b028a0d16"} Sep 07 10:37:46.699477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:46.699367 2565 scope.go:117] "RemoveContainer" containerID="2715620e687748858d2f978088907071e3d563412584746bbc6d8b4b028a0d16" Sep 07 10:37:46.700446 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:46.700419 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:46.700514 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:46.700490 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:46.700557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:46.700525 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:47.707347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:47.707316 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" event={"ID":"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e","Type":"ContainerStarted","Data":"718b2210e8afad149540351b43135a0e046533690896eb2cc0f05d5457ca2912"} Sep 07 10:37:48.220092 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:48.220062 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:48.220280 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:48.220261 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 07 10:37:48.220327 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:48.220307 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 07 10:37:49.220345 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:49.220309 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 07 10:37:49.220756 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:49.220361 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 07 10:37:50.220251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:50.220212 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 07 10:37:50.220517 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:50.220287 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 07 10:37:51.220269 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:51.220231 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 07 10:37:51.220459 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:51.220291 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 07 10:37:52.220124 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:52.220084 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 07 10:37:52.220565 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:52.220145 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 07 10:37:53.223987 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:53.223959 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:53.224462 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:53.224405 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:53.225058 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:53.225040 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:37:56.958216 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.958182 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk"] Sep 07 10:37:56.958686 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.958671 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" Sep 07 10:37:56.958686 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.958686 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="661b9ddb-1092-40f9-a889-5e4107b4d84e" containerName="istio-proxy" Sep 07 10:37:56.958830 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.958728 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" Sep 07 10:37:56.958830 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.958737 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" Sep 07 10:37:56.958830 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.958814 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bd1e1f-6459-4f77-8d03-d448a6e90964" containerName="istio-proxy" Sep 07 10:37:56.962006 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.961987 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:56.964794 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.964773 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-125f28-openshift-default-dockercfg-lzp9r\"" Sep 07 10:37:56.979252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:56.979216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk"] Sep 07 10:37:57.105669 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d80da-5b56-42ac-b508-b01a6e94e4a9-istiod-ca-cert\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.105857 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-token\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.105857 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105746 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-certs\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.105857 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-podinfo\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.105857 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-credential-socket\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.105992 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-kube-api-access-9t8ms\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.105992 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-data\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.105992 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-socket\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.105992 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.105965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-envoy\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206706 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-token\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206706 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-certs\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-podinfo\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-credential-socket\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-kube-api-access-9t8ms\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-data\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-socket\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-envoy\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.206974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.206977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d80da-5b56-42ac-b508-b01a6e94e4a9-istiod-ca-cert\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.207397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.207131 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-certs\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.207397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.207283 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-socket\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.207397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.207332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-credential-socket\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.207397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.207361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-data\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.207674 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.207654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d80da-5b56-42ac-b508-b01a6e94e4a9-istiod-ca-cert\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.209359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.209309 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-envoy\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.209447 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.209382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-podinfo\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.227728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.227705 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-token\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.231811 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.231788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-kube-api-access-9t8ms\") pod \"e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.272370 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.272337 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:57.551258 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.551231 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk"] Sep 07 10:37:57.552802 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:37:57.552777 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964d80da_5b56_42ac_b508_b01a6e94e4a9.slice/crio-5f4231df1a1c6dfc3f24b1fff57e622805b107f5e927f4765f8bc6f3713e34ea WatchSource:0}: Error finding container 5f4231df1a1c6dfc3f24b1fff57e622805b107f5e927f4765f8bc6f3713e34ea: Status 404 returned error can't find the container with id 5f4231df1a1c6dfc3f24b1fff57e622805b107f5e927f4765f8bc6f3713e34ea Sep 07 10:37:57.555024 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.554992 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:57.555097 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.555055 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:57.555097 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.555086 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:37:57.775341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.775305 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" event={"ID":"964d80da-5b56-42ac-b508-b01a6e94e4a9","Type":"ContainerStarted","Data":"5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a"} Sep 07 10:37:57.775514 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.775386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" event={"ID":"964d80da-5b56-42ac-b508-b01a6e94e4a9","Type":"ContainerStarted","Data":"5f4231df1a1c6dfc3f24b1fff57e622805b107f5e927f4765f8bc6f3713e34ea"} Sep 07 10:37:57.800868 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:57.800744 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" podStartSLOduration=1.800722973 podStartE2EDuration="1.800722973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:37:57.799149051 +0000 UTC m=+1599.785429387" watchObservedRunningTime="2026-09-07 10:37:57.800722973 +0000 UTC m=+1599.787003309" Sep 07 10:37:58.272927 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:58.272890 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:58.278730 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:58.278704 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:58.781482 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:58.781444 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:37:58.782430 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:37:58.782412 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:38:17.901680 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:17.901591 2565 generic.go:356] "Generic (PLEG): container finished" podID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerID="5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a" exitCode=0 Sep 07 10:38:17.902090 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:17.901692 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" event={"ID":"964d80da-5b56-42ac-b508-b01a6e94e4a9","Type":"ContainerDied","Data":"5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a"} Sep 07 10:38:17.902187 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:17.902173 2565 scope.go:117] "RemoveContainer" containerID="5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a" Sep 07 10:38:17.903394 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:17.903369 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:38:17.903474 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:17.903439 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:38:17.903516 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:17.903474 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:38:18.909595 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:18.909554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" event={"ID":"964d80da-5b56-42ac-b508-b01a6e94e4a9","Type":"ContainerStarted","Data":"c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a"} Sep 07 10:38:19.273321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:19.273289 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:38:19.273531 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:19.273510 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:38:19.273588 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:19.273569 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 07 10:38:20.273393 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:20.273354 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:38:20.273809 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:20.273413 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 07 10:38:21.273301 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:21.273257 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:38:21.273613 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:21.273321 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 07 10:38:22.273084 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:22.273041 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:38:22.273299 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:22.273109 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 07 10:38:23.273731 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:23.273695 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:38:23.274126 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:23.273764 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 07 10:38:24.314746 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:24.314712 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:38:24.315368 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:24.315347 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:38:24.316055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:24.316028 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:38:29.637277 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:38:29.637247 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jxjws_170e144a-1758-4aa1-9f7f-9130f65ce97a/payload-processing/0.log" Sep 07 10:39:05.475148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:05.475114 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jxjws_170e144a-1758-4aa1-9f7f-9130f65ce97a/payload-processing/0.log" Sep 07 10:39:06.127039 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.127009 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm"] Sep 07 10:39:06.131123 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.131100 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.134553 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.134530 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-c43305-openshift-default-dockercfg-56zkw\"" Sep 07 10:39:06.156436 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.156403 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm"] Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-data\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-socket\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239508 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-envoy\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-token\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-credential-socket\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-podinfo\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjw5c\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-kube-api-access-qjw5c\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.239917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.239763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-certs\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341150 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-token\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-credential-socket\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-podinfo\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341359 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341229 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjw5c\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-kube-api-access-qjw5c\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341504 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-certs\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341504 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-data\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341504 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-socket\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341672 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341672 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-envoy\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-credential-socket\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341803 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341739 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-data\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.341905 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.341812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-certs\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.342042 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.342017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-socket\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.342251 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.342232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.343908 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.343889 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-envoy\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.343974 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.343955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-podinfo\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.353933 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.353909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-token\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.354462 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.354446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjw5c\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-kube-api-access-qjw5c\") pod \"e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.441631 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.441539 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:06.724626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.724597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm"] Sep 07 10:39:06.725702 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:39:06.725676 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afcbfc4_5718_4aad_8aca_6a9f9b73ed11.slice/crio-77a87cc23991f920558e2220d12d251388153defc6daf130a581fba1d7744061 WatchSource:0}: Error finding container 77a87cc23991f920558e2220d12d251388153defc6daf130a581fba1d7744061: Status 404 returned error can't find the container with id 77a87cc23991f920558e2220d12d251388153defc6daf130a581fba1d7744061 Sep 07 10:39:06.728170 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.728125 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:39:06.728241 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.728204 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:39:06.728279 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:06.728252 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:39:07.197250 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:07.197136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" event={"ID":"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11","Type":"ContainerStarted","Data":"3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0"} Sep 07 10:39:07.197250 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:07.197203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" event={"ID":"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11","Type":"ContainerStarted","Data":"77a87cc23991f920558e2220d12d251388153defc6daf130a581fba1d7744061"} Sep 07 10:39:07.220037 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:07.219987 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podStartSLOduration=1.219973268 podStartE2EDuration="1.219973268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:07.217747575 +0000 UTC m=+1669.204027939" watchObservedRunningTime="2026-09-07 10:39:07.219973268 +0000 UTC m=+1669.206253602" Sep 07 10:39:07.442600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:07.442573 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:07.448405 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:07.448329 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:08.203325 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:08.203293 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:08.204321 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:08.204301 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:18.949378 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:18.949346 2565 scope.go:117] "RemoveContainer" containerID="be0937c2760057ee2f52c68e806e2299154001432eb0b21115f58450418c1762" Sep 07 10:39:28.326114 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:28.326078 2565 generic.go:356] "Generic (PLEG): container finished" podID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerID="3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0" exitCode=0 Sep 07 10:39:28.326568 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:28.326155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" event={"ID":"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11","Type":"ContainerDied","Data":"3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0"} Sep 07 10:39:28.326646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:28.326630 2565 scope.go:117] "RemoveContainer" containerID="3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0" Sep 07 10:39:28.327786 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:28.327759 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:39:28.327883 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:28.327869 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:39:28.327927 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:28.327904 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 07 10:39:29.334239 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:29.334202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" event={"ID":"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11","Type":"ContainerStarted","Data":"4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2"} Sep 07 10:39:29.442145 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:29.442111 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:29.442399 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:29.442376 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 07 10:39:29.442457 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:29.442428 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 07 10:39:30.442502 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:30.442461 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 07 10:39:30.443115 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:30.443085 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 07 10:39:31.441979 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:31.441942 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 07 10:39:31.442211 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:31.442007 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 07 10:39:32.442320 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:32.442279 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 07 10:39:32.442818 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:32.442349 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 07 10:39:33.442432 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:33.442396 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 07 10:39:33.443018 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:33.442988 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 07 10:39:34.442276 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.442236 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 07 10:39:34.442589 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.442561 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 07 10:39:34.721904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.721873 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl"] Sep 07 10:39:34.725638 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.725623 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" Sep 07 10:39:34.728391 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.728373 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:39:34.728454 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.728414 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:39:34.740182 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.737597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl"] Sep 07 10:39:34.853229 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.853189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc523f-3211-40f8-bd8f-3f549fa559aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl\" (UID: \"c9dc523f-3211-40f8-bd8f-3f549fa559aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" Sep 07 10:39:34.954802 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.954766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc523f-3211-40f8-bd8f-3f549fa559aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl\" (UID: \"c9dc523f-3211-40f8-bd8f-3f549fa559aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" Sep 07 10:39:34.955493 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:34.955473 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc523f-3211-40f8-bd8f-3f549fa559aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl\" (UID: \"c9dc523f-3211-40f8-bd8f-3f549fa559aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" Sep 07 10:39:35.045782 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:35.045691 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" Sep 07 10:39:35.331817 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:35.331700 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl"] Sep 07 10:39:35.334343 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:39:35.334316 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dc523f_3211_40f8_bd8f_3f549fa559aa.slice/crio-bc32d41d1b47773ad9d57c3c538d53639170e15ad45278b38d2d50c89a0d16c5 WatchSource:0}: Error finding container bc32d41d1b47773ad9d57c3c538d53639170e15ad45278b38d2d50c89a0d16c5: Status 404 returned error can't find the container with id bc32d41d1b47773ad9d57c3c538d53639170e15ad45278b38d2d50c89a0d16c5 Sep 07 10:39:35.377725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:35.377699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" event={"ID":"c9dc523f-3211-40f8-bd8f-3f549fa559aa","Type":"ContainerStarted","Data":"bc32d41d1b47773ad9d57c3c538d53639170e15ad45278b38d2d50c89a0d16c5"} Sep 07 10:39:35.446305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:35.446278 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:35.446668 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:35.446576 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:35.447318 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:35.447301 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:36.384892 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:36.384787 2565 generic.go:356] "Generic (PLEG): container finished" podID="c9dc523f-3211-40f8-bd8f-3f549fa559aa" containerID="a97fbf9b227dc33df10d850a3172cef0e28aee125f447fb2499bfbd83c3c02ba" exitCode=0 Sep 07 10:39:36.385069 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:36.384884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" event={"ID":"c9dc523f-3211-40f8-bd8f-3f549fa559aa","Type":"ContainerDied","Data":"a97fbf9b227dc33df10d850a3172cef0e28aee125f447fb2499bfbd83c3c02ba"} Sep 07 10:39:37.526775 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:37.526752 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" Sep 07 10:39:37.583981 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:37.583941 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc523f-3211-40f8-bd8f-3f549fa559aa-maas-api-service-ca\") pod \"c9dc523f-3211-40f8-bd8f-3f549fa559aa\" (UID: \"c9dc523f-3211-40f8-bd8f-3f549fa559aa\") " Sep 07 10:39:37.584334 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:37.584312 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9dc523f-3211-40f8-bd8f-3f549fa559aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9dc523f-3211-40f8-bd8f-3f549fa559aa" (UID: "c9dc523f-3211-40f8-bd8f-3f549fa559aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:39:37.584444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:37.584420 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9dc523f-3211-40f8-bd8f-3f549fa559aa-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:38.403059 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:38.403023 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" event={"ID":"c9dc523f-3211-40f8-bd8f-3f549fa559aa","Type":"ContainerDied","Data":"bc32d41d1b47773ad9d57c3c538d53639170e15ad45278b38d2d50c89a0d16c5"} Sep 07 10:39:38.403267 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:38.403107 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc32d41d1b47773ad9d57c3c538d53639170e15ad45278b38d2d50c89a0d16c5" Sep 07 10:39:38.403267 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:38.403035 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl" Sep 07 10:39:41.132443 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:41.132354 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl"] Sep 07 10:39:41.134602 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:41.134579 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c43305-7htpl"] Sep 07 10:39:42.544487 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:42.544455 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9dc523f-3211-40f8-bd8f-3f549fa559aa" path="/var/lib/kubelet/pods/c9dc523f-3211-40f8-bd8f-3f549fa559aa/volumes" Sep 07 10:39:47.892090 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:47.892052 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm"] Sep 07 10:39:47.892529 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:47.892300 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" containerID="cri-o://4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2" gracePeriod=30 Sep 07 10:39:50.448622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:50.448584 2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 10:39:50.449024 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:50.448669 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:39:51.403369 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.403333 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf"] Sep 07 10:39:51.403852 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.403835 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9dc523f-3211-40f8-bd8f-3f549fa559aa" containerName="revoke-keys" Sep 07 10:39:51.403947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.403856 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9dc523f-3211-40f8-bd8f-3f549fa559aa" containerName="revoke-keys" Sep 07 10:39:51.404003 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.403971 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9dc523f-3211-40f8-bd8f-3f549fa559aa" containerName="revoke-keys" Sep 07 10:39:51.407023 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.407003 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" Sep 07 10:39:51.409889 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.409863 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:39:51.409994 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.409894 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:39:51.415730 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.415701 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf"] Sep 07 10:39:51.527558 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.527523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a46952-8db3-4199-86df-f214baef801a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf\" (UID: \"61a46952-8db3-4199-86df-f214baef801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" Sep 07 10:39:51.628515 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.628477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a46952-8db3-4199-86df-f214baef801a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf\" (UID: \"61a46952-8db3-4199-86df-f214baef801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" Sep 07 10:39:51.629148 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.629128 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a46952-8db3-4199-86df-f214baef801a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf\" (UID: \"61a46952-8db3-4199-86df-f214baef801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" Sep 07 10:39:51.718096 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:51.718006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" Sep 07 10:39:52.009254 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:52.009044 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf"] Sep 07 10:39:52.012090 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:39:52.012061 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a46952_8db3_4199_86df_f214baef801a.slice/crio-2b65a093212109680e5648cca4550cedba248e87ddba2cbc32fad792074a4733 WatchSource:0}: Error finding container 2b65a093212109680e5648cca4550cedba248e87ddba2cbc32fad792074a4733: Status 404 returned error can't find the container with id 2b65a093212109680e5648cca4550cedba248e87ddba2cbc32fad792074a4733 Sep 07 10:39:52.482635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:52.482601 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" event={"ID":"61a46952-8db3-4199-86df-f214baef801a","Type":"ContainerStarted","Data":"9d99e14695d4f6fbaf31ff4f83ddc248015470cd465f9602256489c26d861835"} Sep 07 10:39:52.482829 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:52.482660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" event={"ID":"61a46952-8db3-4199-86df-f214baef801a","Type":"ContainerStarted","Data":"2b65a093212109680e5648cca4550cedba248e87ddba2cbc32fad792074a4733"} Sep 07 10:39:52.501116 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:52.501062 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" podStartSLOduration=1.501044967 podStartE2EDuration="1.501044967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:52.497818609 +0000 UTC m=+1714.484098942" watchObservedRunningTime="2026-09-07 10:39:52.501044967 +0000 UTC m=+1714.487325302" Sep 07 10:39:53.144943 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.144919 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:53.248526 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248494 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-data\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.248718 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjw5c\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-kube-api-access-qjw5c\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.248718 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-credential-socket\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.248718 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248643 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istiod-ca-cert\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.248718 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248666 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-envoy\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.248925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248791 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-data" (OuterVolumeSpecName: "istio-data") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:53.248925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248799 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-certs\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.248925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248835 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-token\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.248925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248869 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-socket\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.248925 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:53.249224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.248935 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-podinfo\") pod \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\" (UID: \"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11\") " Sep 07 10:39:53.249224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.249002 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:39:53.249224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.249041 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:53.249224 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.249120 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:53.249459 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.249360 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.249459 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.249379 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.249459 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.249402 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.249459 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.249415 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.249459 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.249427 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.250986 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.250963 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-kube-api-access-qjw5c" (OuterVolumeSpecName: "kube-api-access-qjw5c") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "kube-api-access-qjw5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:39:53.250986 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.250983 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-token" (OuterVolumeSpecName: "istio-token") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:39:53.251152 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.251063 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:39:53.251267 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.251252 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" (UID: "7afcbfc4-5718-4aad-8aca-6a9f9b73ed11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:53.350360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.350327 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.350360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.350356 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjw5c\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-kube-api-access-qjw5c\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.350360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.350366 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.350589 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.350378 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:53.490304 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.490266 2565 generic.go:356] "Generic (PLEG): container finished" podID="61a46952-8db3-4199-86df-f214baef801a" containerID="9d99e14695d4f6fbaf31ff4f83ddc248015470cd465f9602256489c26d861835" exitCode=0 Sep 07 10:39:53.490498 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.490389 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" event={"ID":"61a46952-8db3-4199-86df-f214baef801a","Type":"ContainerDied","Data":"9d99e14695d4f6fbaf31ff4f83ddc248015470cd465f9602256489c26d861835"} Sep 07 10:39:53.492255 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.492231 2565 generic.go:356] "Generic (PLEG): container finished" podID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerID="4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2" exitCode=0 Sep 07 10:39:53.492378 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.492274 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" Sep 07 10:39:53.492378 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.492289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" event={"ID":"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11","Type":"ContainerDied","Data":"4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2"} Sep 07 10:39:53.492378 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.492325 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm" event={"ID":"7afcbfc4-5718-4aad-8aca-6a9f9b73ed11","Type":"ContainerDied","Data":"77a87cc23991f920558e2220d12d251388153defc6daf130a581fba1d7744061"} Sep 07 10:39:53.492378 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.492348 2565 scope.go:117] "RemoveContainer" containerID="4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2" Sep 07 10:39:53.503564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.503545 2565 scope.go:117] "RemoveContainer" containerID="3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0" Sep 07 10:39:53.515804 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.515771 2565 scope.go:117] "RemoveContainer" containerID="4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2" Sep 07 10:39:53.516359 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:39:53.516339 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2\": container with ID starting with 4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2 not found: ID does not exist" containerID="4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2" Sep 07 10:39:53.516431 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.516369 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2"} err="failed to get container status \"4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2\": rpc error: code = NotFound desc = could not find container \"4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2\": container with ID starting with 4b077d797a5430e4759423362170e0a1226a504ecc07c807c629630c26887bd2 not found: ID does not exist" Sep 07 10:39:53.516431 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.516388 2565 scope.go:117] "RemoveContainer" containerID="3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0" Sep 07 10:39:53.516688 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:39:53.516666 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0\": container with ID starting with 3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0 not found: ID does not exist" containerID="3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0" Sep 07 10:39:53.516783 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.516685 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0"} err="failed to get container status \"3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0\": rpc error: code = NotFound desc = could not find container \"3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0\": container with ID starting with 3681ce20d7d132a204a3d3a0aaf556c006df7082ee98cc8e8284fc8bc48a2ae0 not found: ID does not exist" Sep 07 10:39:53.522559 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.522532 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm"] Sep 07 10:39:53.526935 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:53.526911 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c43305-openshift-default-5ccddbbbc6-9s4nm"] Sep 07 10:39:54.550190 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:54.550129 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" path="/var/lib/kubelet/pods/7afcbfc4-5718-4aad-8aca-6a9f9b73ed11/volumes" Sep 07 10:39:54.638896 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:54.638874 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" Sep 07 10:39:54.764458 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:54.764353 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a46952-8db3-4199-86df-f214baef801a-maas-api-service-ca\") pod \"61a46952-8db3-4199-86df-f214baef801a\" (UID: \"61a46952-8db3-4199-86df-f214baef801a\") " Sep 07 10:39:54.764768 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:54.764744 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a46952-8db3-4199-86df-f214baef801a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61a46952-8db3-4199-86df-f214baef801a" (UID: "61a46952-8db3-4199-86df-f214baef801a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:39:54.865256 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:54.865211 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a46952-8db3-4199-86df-f214baef801a-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:39:55.505959 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:55.505923 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" event={"ID":"61a46952-8db3-4199-86df-f214baef801a","Type":"ContainerDied","Data":"2b65a093212109680e5648cca4550cedba248e87ddba2cbc32fad792074a4733"} Sep 07 10:39:55.505959 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:55.505950 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf" Sep 07 10:39:55.506203 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:39:55.505973 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b65a093212109680e5648cca4550cedba248e87ddba2cbc32fad792074a4733" Sep 07 10:40:07.125256 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:07.125175 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf"] Sep 07 10:40:07.127909 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:07.127883 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-92aa2e-ggkqf"] Sep 07 10:40:08.544130 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:08.544083 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a46952-8db3-4199-86df-f214baef801a" path="/var/lib/kubelet/pods/61a46952-8db3-4199-86df-f214baef801a/volumes" Sep 07 10:40:55.756884 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:55.756852 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn"] Sep 07 10:40:55.757488 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:55.757060 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" containerID="cri-o://718b2210e8afad149540351b43135a0e046533690896eb2cc0f05d5457ca2912" gracePeriod=30 Sep 07 10:40:56.128917 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.128836 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89"] Sep 07 10:40:56.129259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129246 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a46952-8db3-4199-86df-f214baef801a" containerName="revoke-keys" Sep 07 10:40:56.129319 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129260 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a46952-8db3-4199-86df-f214baef801a" containerName="revoke-keys" Sep 07 10:40:56.129319 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129283 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" Sep 07 10:40:56.129319 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129288 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" Sep 07 10:40:56.129319 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129297 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" Sep 07 10:40:56.129319 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129302 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" Sep 07 10:40:56.129477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129367 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" Sep 07 10:40:56.129477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129375 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a46952-8db3-4199-86df-f214baef801a" containerName="revoke-keys" Sep 07 10:40:56.129477 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.129385 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afcbfc4-5718-4aad-8aca-6a9f9b73ed11" containerName="istio-proxy" Sep 07 10:40:56.132614 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.132598 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" Sep 07 10:40:56.135348 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.135323 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:40:56.135490 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.135323 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:40:56.140754 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.140596 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89"] Sep 07 10:40:56.186546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.186509 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ce8a26-7625-496d-9b07-caafad19e138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89\" (UID: \"c0ce8a26-7625-496d-9b07-caafad19e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" Sep 07 10:40:56.287828 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.287793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ce8a26-7625-496d-9b07-caafad19e138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89\" (UID: \"c0ce8a26-7625-496d-9b07-caafad19e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" Sep 07 10:40:56.288851 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.288816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ce8a26-7625-496d-9b07-caafad19e138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89\" (UID: \"c0ce8a26-7625-496d-9b07-caafad19e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" Sep 07 10:40:56.444761 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.444677 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" Sep 07 10:40:56.729193 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.729149 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89"] Sep 07 10:40:56.730764 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:40:56.730738 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ce8a26_7625_496d_9b07_caafad19e138.slice/crio-1d17fc99d345a04bc31023ed42f63895a2f80ee5b776af058beb6cc21427998b WatchSource:0}: Error finding container 1d17fc99d345a04bc31023ed42f63895a2f80ee5b776af058beb6cc21427998b: Status 404 returned error can't find the container with id 1d17fc99d345a04bc31023ed42f63895a2f80ee5b776af058beb6cc21427998b Sep 07 10:40:56.853449 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:56.853414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" event={"ID":"c0ce8a26-7625-496d-9b07-caafad19e138","Type":"ContainerStarted","Data":"1d17fc99d345a04bc31023ed42f63895a2f80ee5b776af058beb6cc21427998b"} Sep 07 10:40:57.860144 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:57.860056 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0ce8a26-7625-496d-9b07-caafad19e138" containerID="947df8ca2f9f0f84d28481e72ac55020cd022a7a39bbc5c2d20707f05b8c4a88" exitCode=0 Sep 07 10:40:57.860144 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:57.860090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" event={"ID":"c0ce8a26-7625-496d-9b07-caafad19e138","Type":"ContainerDied","Data":"947df8ca2f9f0f84d28481e72ac55020cd022a7a39bbc5c2d20707f05b8c4a88"} Sep 07 10:40:59.005551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:59.005528 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" Sep 07 10:40:59.117874 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:59.117771 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ce8a26-7625-496d-9b07-caafad19e138-maas-api-service-ca\") pod \"c0ce8a26-7625-496d-9b07-caafad19e138\" (UID: \"c0ce8a26-7625-496d-9b07-caafad19e138\") " Sep 07 10:40:59.118179 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:59.118136 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ce8a26-7625-496d-9b07-caafad19e138-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0ce8a26-7625-496d-9b07-caafad19e138" (UID: "c0ce8a26-7625-496d-9b07-caafad19e138"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:40:59.219300 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:59.219269 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ce8a26-7625-496d-9b07-caafad19e138-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:40:59.872841 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:59.872806 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" event={"ID":"c0ce8a26-7625-496d-9b07-caafad19e138","Type":"ContainerDied","Data":"1d17fc99d345a04bc31023ed42f63895a2f80ee5b776af058beb6cc21427998b"} Sep 07 10:40:59.873014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:59.872879 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d17fc99d345a04bc31023ed42f63895a2f80ee5b776af058beb6cc21427998b" Sep 07 10:40:59.873014 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:40:59.872820 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89" Sep 07 10:41:00.880479 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:00.880451 2565 generic.go:356] "Generic (PLEG): container finished" podID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerID="718b2210e8afad149540351b43135a0e046533690896eb2cc0f05d5457ca2912" exitCode=0 Sep 07 10:41:00.880834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:00.880523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" event={"ID":"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e","Type":"ContainerDied","Data":"718b2210e8afad149540351b43135a0e046533690896eb2cc0f05d5457ca2912"} Sep 07 10:41:00.880834 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:00.880584 2565 scope.go:117] "RemoveContainer" containerID="2715620e687748858d2f978088907071e3d563412584746bbc6d8b4b028a0d16" Sep 07 10:41:01.011273 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.011252 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:41:01.037259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-podinfo\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037259 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037240 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-envoy\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037268 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-data\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037311 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-token\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037354 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-credential-socket\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037389 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-certs\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037483 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037414 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqcs6\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-kube-api-access-hqcs6\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037520 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istiod-ca-cert\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037546 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-socket\") pod \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\" (UID: \"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e\") " Sep 07 10:41:01.037752 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037725 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:01.038145 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.037965 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.038300 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.038222 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-data" (OuterVolumeSpecName: "istio-data") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:01.038450 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.038414 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:01.038450 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.038410 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:01.038597 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.038509 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:01.039904 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.039860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:01.040128 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.040105 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:01.040258 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.040137 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:41:01.040354 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.040334 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-kube-api-access-hqcs6" (OuterVolumeSpecName: "kube-api-access-hqcs6") pod "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" (UID: "97a47ee8-6be1-4b0e-93f5-fa4fda879d5e"). InnerVolumeSpecName "kube-api-access-hqcs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:01.139252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.139207 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.139252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.139244 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.139252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.139252 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.139252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.139260 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.139531 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.139269 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqcs6\" (UniqueName: \"kubernetes.io/projected/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-kube-api-access-hqcs6\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.139531 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.139278 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.139531 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.139287 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.139531 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.139294 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:01.888364 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.888331 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" Sep 07 10:41:01.888893 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.888371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn" event={"ID":"97a47ee8-6be1-4b0e-93f5-fa4fda879d5e","Type":"ContainerDied","Data":"05f23ee1862799c82018b62d0a8af9ba45ad3bbaf374d7e2b0f34bfbb5e9fdec"} Sep 07 10:41:01.888893 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.888449 2565 scope.go:117] "RemoveContainer" containerID="718b2210e8afad149540351b43135a0e046533690896eb2cc0f05d5457ca2912" Sep 07 10:41:01.919376 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.919344 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn"] Sep 07 10:41:01.927235 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:01.927206 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-92aa2e-openshift-default-6c56c5487d-bqmgn"] Sep 07 10:41:02.545331 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:02.545299 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" path="/var/lib/kubelet/pods/97a47ee8-6be1-4b0e-93f5-fa4fda879d5e/volumes" Sep 07 10:41:06.927326 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:06.927290 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89"] Sep 07 10:41:06.932371 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:06.932346 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-125f28-nhn89"] Sep 07 10:41:08.543951 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:08.543918 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ce8a26-7625-496d-9b07-caafad19e138" path="/var/lib/kubelet/pods/c0ce8a26-7625-496d-9b07-caafad19e138/volumes" Sep 07 10:41:13.334702 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.334617 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk"] Sep 07 10:41:13.335213 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.334923 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" containerID="cri-o://c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a" gracePeriod=30 Sep 07 10:41:13.692600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.692512 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq"] Sep 07 10:41:13.692947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.692934 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" Sep 07 10:41:13.692991 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.692948 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" Sep 07 10:41:13.692991 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.692956 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ce8a26-7625-496d-9b07-caafad19e138" containerName="revoke-keys" Sep 07 10:41:13.692991 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.692963 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ce8a26-7625-496d-9b07-caafad19e138" containerName="revoke-keys" Sep 07 10:41:13.693087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.693041 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" Sep 07 10:41:13.693087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.693052 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" Sep 07 10:41:13.693087 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.693058 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ce8a26-7625-496d-9b07-caafad19e138" containerName="revoke-keys" Sep 07 10:41:13.698046 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.698025 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" Sep 07 10:41:13.700650 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.700622 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:41:13.700795 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.700714 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\"" Sep 07 10:41:13.704643 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.704618 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq"] Sep 07 10:41:13.766989 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.766954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0211dc27-21c6-418e-afe7-419ae477594a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq\" (UID: \"0211dc27-21c6-418e-afe7-419ae477594a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" Sep 07 10:41:13.868070 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.868034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0211dc27-21c6-418e-afe7-419ae477594a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq\" (UID: \"0211dc27-21c6-418e-afe7-419ae477594a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" Sep 07 10:41:13.868714 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:13.868693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0211dc27-21c6-418e-afe7-419ae477594a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq\" (UID: \"0211dc27-21c6-418e-afe7-419ae477594a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" Sep 07 10:41:14.010707 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:14.010670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" Sep 07 10:41:14.286401 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:14.286371 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq"] Sep 07 10:41:14.288596 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:41:14.288567 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0211dc27_21c6_418e_afe7_419ae477594a.slice/crio-2ae291fee0d99143b9da3104d19793908c05db5ce78cb72a68f30330898a31f4 WatchSource:0}: Error finding container 2ae291fee0d99143b9da3104d19793908c05db5ce78cb72a68f30330898a31f4: Status 404 returned error can't find the container with id 2ae291fee0d99143b9da3104d19793908c05db5ce78cb72a68f30330898a31f4 Sep 07 10:41:14.967034 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:14.967006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" event={"ID":"0211dc27-21c6-418e-afe7-419ae477594a","Type":"ContainerStarted","Data":"97d0a2a202f24420c15ce2a96737e1bd190e79957b5987ebfc7435a7799555ac"} Sep 07 10:41:14.967387 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:14.967052 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" event={"ID":"0211dc27-21c6-418e-afe7-419ae477594a","Type":"ContainerStarted","Data":"2ae291fee0d99143b9da3104d19793908c05db5ce78cb72a68f30330898a31f4"} Sep 07 10:41:15.974764 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:15.974732 2565 generic.go:356] "Generic (PLEG): container finished" podID="0211dc27-21c6-418e-afe7-419ae477594a" containerID="97d0a2a202f24420c15ce2a96737e1bd190e79957b5987ebfc7435a7799555ac" exitCode=0 Sep 07 10:41:15.975234 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:15.974817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" event={"ID":"0211dc27-21c6-418e-afe7-419ae477594a","Type":"ContainerDied","Data":"97d0a2a202f24420c15ce2a96737e1bd190e79957b5987ebfc7435a7799555ac"} Sep 07 10:41:16.113143 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:16.113116 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" Sep 07 10:41:16.194055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:16.194009 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0211dc27-21c6-418e-afe7-419ae477594a-maas-api-service-ca\") pod \"0211dc27-21c6-418e-afe7-419ae477594a\" (UID: \"0211dc27-21c6-418e-afe7-419ae477594a\") " Sep 07 10:41:16.194444 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:16.194423 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0211dc27-21c6-418e-afe7-419ae477594a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0211dc27-21c6-418e-afe7-419ae477594a" (UID: "0211dc27-21c6-418e-afe7-419ae477594a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:16.295360 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:16.295275 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0211dc27-21c6-418e-afe7-419ae477594a-maas-api-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:16.983151 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:16.983117 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" Sep 07 10:41:16.983562 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:16.983152 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq" event={"ID":"0211dc27-21c6-418e-afe7-419ae477594a","Type":"ContainerDied","Data":"2ae291fee0d99143b9da3104d19793908c05db5ce78cb72a68f30330898a31f4"} Sep 07 10:41:16.983562 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:16.983219 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ae291fee0d99143b9da3104d19793908c05db5ce78cb72a68f30330898a31f4" Sep 07 10:41:18.598341 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.598278 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:41:18.677278 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.677252 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:41:18.682581 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.682557 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:41:18.720690 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720660 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-socket\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.720890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720698 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-kube-api-access-9t8ms\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.720890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720743 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-token\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.720890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720770 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-certs\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.720890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-data\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.720890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720852 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d80da-5b56-42ac-b508-b01a6e94e4a9-istiod-ca-cert\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.720890 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720880 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-envoy\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.721242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720922 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-credential-socket\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.721242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.720974 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:18.721242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721124 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:18.721242 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721131 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:18.721463 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721273 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-podinfo\") pod \"964d80da-5b56-42ac-b508-b01a6e94e4a9\" (UID: \"964d80da-5b56-42ac-b508-b01a6e94e4a9\") " Sep 07 10:41:18.721463 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721277 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:18.721463 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721343 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964d80da-5b56-42ac-b508-b01a6e94e4a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:18.721626 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721611 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:18.721703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721629 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:18.721703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721641 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:18.721703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721652 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:18.721703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.721668 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d80da-5b56-42ac-b508-b01a6e94e4a9-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:18.723305 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.723271 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:18.723437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.723360 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:18.723437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.723410 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:41:18.723437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.723419 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-kube-api-access-9t8ms" (OuterVolumeSpecName: "kube-api-access-9t8ms") pod "964d80da-5b56-42ac-b508-b01a6e94e4a9" (UID: "964d80da-5b56-42ac-b508-b01a6e94e4a9"). InnerVolumeSpecName "kube-api-access-9t8ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:18.822382 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.822345 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t8ms\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-kube-api-access-9t8ms\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:18.822382 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.822378 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:18.822382 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.822387 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:18.822382 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:18.822395 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d80da-5b56-42ac-b508-b01a6e94e4a9-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:19.002595 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.002561 2565 generic.go:356] "Generic (PLEG): container finished" podID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerID="c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a" exitCode=0 Sep 07 10:41:19.002725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.002614 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" Sep 07 10:41:19.002725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.002639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" event={"ID":"964d80da-5b56-42ac-b508-b01a6e94e4a9","Type":"ContainerDied","Data":"c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a"} Sep 07 10:41:19.002725 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.002718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk" event={"ID":"964d80da-5b56-42ac-b508-b01a6e94e4a9","Type":"ContainerDied","Data":"5f4231df1a1c6dfc3f24b1fff57e622805b107f5e927f4765f8bc6f3713e34ea"} Sep 07 10:41:19.002826 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.002741 2565 scope.go:117] "RemoveContainer" containerID="c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a" Sep 07 10:41:19.012557 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.012533 2565 scope.go:117] "RemoveContainer" containerID="5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a" Sep 07 10:41:19.021536 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.021514 2565 scope.go:117] "RemoveContainer" containerID="c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a" Sep 07 10:41:19.021837 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:41:19.021815 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a\": container with ID starting with c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a not found: ID does not exist" containerID="c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a" Sep 07 10:41:19.021913 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.021847 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a"} err="failed to get container status \"c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a\": rpc error: code = NotFound desc = could not find container \"c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a\": container with ID starting with c930554a74b380da63426a054d1e1cc3b59885045eb5f7a81fff10397e64634a not found: ID does not exist" Sep 07 10:41:19.021913 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.021865 2565 scope.go:117] "RemoveContainer" containerID="5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a" Sep 07 10:41:19.022132 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:41:19.022112 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a\": container with ID starting with 5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a not found: ID does not exist" containerID="5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a" Sep 07 10:41:19.022189 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.022137 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a"} err="failed to get container status \"5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a\": rpc error: code = NotFound desc = could not find container \"5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a\": container with ID starting with 5ec26f25ea1f14c6ab0ffc07198b44ca6485a4b05f5cb15fe2aba5e99b16a79a not found: ID does not exist" Sep 07 10:41:19.026467 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.026443 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk"] Sep 07 10:41:19.031094 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:19.031065 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-125f28-openshift-default-885c5b98d-pr2wk"] Sep 07 10:41:20.544641 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:20.544590 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" path="/var/lib/kubelet/pods/964d80da-5b56-42ac-b508-b01a6e94e4a9/volumes" Sep 07 10:41:24.358664 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:24.358633 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z"] Sep 07 10:41:24.360739 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:24.360714 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0a577c-29812950-zkf2z"] Sep 07 10:41:24.544635 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:24.544604 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb34e07-90de-4fdb-ab28-a6902bf866d1" path="/var/lib/kubelet/pods/bdb34e07-90de-4fdb-ab28-a6902bf866d1/volumes" Sep 07 10:41:24.616602 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:24.616493 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq"] Sep 07 10:41:24.619896 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:24.619852 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0a577c-hmggq"] Sep 07 10:41:26.544426 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:26.544385 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0211dc27-21c6-418e-afe7-419ae477594a" path="/var/lib/kubelet/pods/0211dc27-21c6-418e-afe7-419ae477594a/volumes" Sep 07 10:41:31.417845 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:31.417805 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7"] Sep 07 10:41:31.418339 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:31.418069 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" containerID="cri-o://98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b" gracePeriod=30 Sep 07 10:41:31.790386 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:31.790343 2565 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 10:41:31.790567 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:31.790400 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:41:36.669252 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.669226 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:41:36.691404 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-envoy\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.691404 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691366 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-credential-socket\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.691622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691418 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-token\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.691622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691455 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b52e926-8672-4aa0-9402-2492fd59dc88-istiod-ca-cert\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.691622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691473 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnvn2\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-kube-api-access-gnvn2\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.691622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691491 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-socket\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.691622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691507 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-certs\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.691622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691525 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-data\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.691622 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691563 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-podinfo\") pod \"0b52e926-8672-4aa0-9402-2492fd59dc88\" (UID: \"0b52e926-8672-4aa0-9402-2492fd59dc88\") " Sep 07 10:41:36.692398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691765 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:36.692398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691793 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:36.692398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691861 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b52e926-8672-4aa0-9402-2492fd59dc88-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:36.692398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691913 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:36.692398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.691943 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:36.692398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.692118 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:36.692398 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.692148 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:36.694552 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.694357 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:41:36.694666 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.694606 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:36.695019 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.694986 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-kube-api-access-gnvn2" (OuterVolumeSpecName: "kube-api-access-gnvn2") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "kube-api-access-gnvn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:36.695211 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.695024 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b52e926-8672-4aa0-9402-2492fd59dc88" (UID: "0b52e926-8672-4aa0-9402-2492fd59dc88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:36.793302 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.793261 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:36.793302 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.793293 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b52e926-8672-4aa0-9402-2492fd59dc88-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:36.793302 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.793304 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnvn2\" (UniqueName: \"kubernetes.io/projected/0b52e926-8672-4aa0-9402-2492fd59dc88-kube-api-access-gnvn2\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:36.793546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.793313 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:36.793546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.793323 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:36.793546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.793330 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:36.793546 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:36.793338 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b52e926-8672-4aa0-9402-2492fd59dc88-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:37.109570 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.109537 2565 generic.go:356] "Generic (PLEG): container finished" podID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerID="98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b" exitCode=0 Sep 07 10:41:37.109777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.109597 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" Sep 07 10:41:37.109777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.109620 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" event={"ID":"0b52e926-8672-4aa0-9402-2492fd59dc88","Type":"ContainerDied","Data":"98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b"} Sep 07 10:41:37.109777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.109690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7" event={"ID":"0b52e926-8672-4aa0-9402-2492fd59dc88","Type":"ContainerDied","Data":"10c9e048726174d0b941f8fe580c971ad369e51b34403b30f04991a7615332a8"} Sep 07 10:41:37.109777 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.109717 2565 scope.go:117] "RemoveContainer" containerID="98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b" Sep 07 10:41:37.127827 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.127806 2565 scope.go:117] "RemoveContainer" containerID="75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1" Sep 07 10:41:37.136328 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.136305 2565 scope.go:117] "RemoveContainer" containerID="98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b" Sep 07 10:41:37.136619 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:41:37.136598 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b\": container with ID starting with 98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b not found: ID does not exist" containerID="98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b" Sep 07 10:41:37.136703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.136627 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b"} err="failed to get container status \"98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b\": rpc error: code = NotFound desc = could not find container \"98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b\": container with ID starting with 98c90b8bc3732f6976c424468a0e242cb6cfa6e1b03d431d93c1c74c41c0094b not found: ID does not exist" Sep 07 10:41:37.136703 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.136645 2565 scope.go:117] "RemoveContainer" containerID="75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1" Sep 07 10:41:37.136904 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:41:37.136891 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1\": container with ID starting with 75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1 not found: ID does not exist" containerID="75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1" Sep 07 10:41:37.136947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.136906 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1"} err="failed to get container status \"75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1\": rpc error: code = NotFound desc = could not find container \"75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1\": container with ID starting with 75b67231539b4dd9588f6ccae1569305b7aba73c374a23a558f6898208d94bb1 not found: ID does not exist" Sep 07 10:41:37.141035 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.141011 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7"] Sep 07 10:41:37.145412 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:37.145386 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0a577c-openshift-default-869576978b-4z2z7"] Sep 07 10:41:38.544473 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:38.544442 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" path="/var/lib/kubelet/pods/0b52e926-8672-4aa0-9402-2492fd59dc88/volumes" Sep 07 10:41:42.656551 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:42.656520 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62"] Sep 07 10:41:42.662817 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:42.662787 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-93e41e-29812950-4sx62"] Sep 07 10:41:44.544679 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:44.544643 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="080e0ecb-3646-4836-b815-6ef146b19137" path="/var/lib/kubelet/pods/080e0ecb-3646-4836-b815-6ef146b19137/volumes" Sep 07 10:41:50.195426 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:50.188971 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp"] Sep 07 10:41:50.195426 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:50.189392 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" containerID="cri-o://37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74" gracePeriod=30 Sep 07 10:41:55.464149 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.464079 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:41:55.592298 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592259 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-certs\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.592489 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592332 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-socket\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.592489 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592369 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918e9d-81f9-4f8f-9905-f4e91f738770-istiod-ca-cert\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.592489 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592433 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-token\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.592489 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592481 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-data\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.592699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-credential-socket\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.592699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592587 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:55.592699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592641 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:55.592699 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592646 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-podinfo\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.592914 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592735 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zqvz\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-kube-api-access-8zqvz\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.592914 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592773 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:55.592914 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.592783 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-envoy\") pod \"1a918e9d-81f9-4f8f-9905-f4e91f738770\" (UID: \"1a918e9d-81f9-4f8f-9905-f4e91f738770\") " Sep 07 10:41:55.593207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.593133 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:55.593295 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.593227 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-credential-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:55.593295 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.593249 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:55.593295 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.593265 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-workload-socket\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:55.593437 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.593317 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a918e9d-81f9-4f8f-9905-f4e91f738770-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:55.594970 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.594948 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-kube-api-access-8zqvz" (OuterVolumeSpecName: "kube-api-access-8zqvz") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "kube-api-access-8zqvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:55.595400 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.595382 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:41:55.595490 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.595393 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:55.595490 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.595463 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a918e9d-81f9-4f8f-9905-f4e91f738770" (UID: "1a918e9d-81f9-4f8f-9905-f4e91f738770"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:55.694508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.694467 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-token\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:55.694508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.694495 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-data\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:55.694508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.694505 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-podinfo\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:55.694508 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.694514 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqvz\" (UniqueName: \"kubernetes.io/projected/1a918e9d-81f9-4f8f-9905-f4e91f738770-kube-api-access-8zqvz\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:55.694971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.694528 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a918e9d-81f9-4f8f-9905-f4e91f738770-istio-envoy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:55.694971 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:55.694541 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a918e9d-81f9-4f8f-9905-f4e91f738770-istiod-ca-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\"" Sep 07 10:41:56.220378 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.220347 2565 generic.go:356] "Generic (PLEG): container finished" podID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerID="37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74" exitCode=0 Sep 07 10:41:56.220570 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.220399 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" Sep 07 10:41:56.220570 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.220435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" event={"ID":"1a918e9d-81f9-4f8f-9905-f4e91f738770","Type":"ContainerDied","Data":"37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74"} Sep 07 10:41:56.220570 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.220490 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp" event={"ID":"1a918e9d-81f9-4f8f-9905-f4e91f738770","Type":"ContainerDied","Data":"ea8a5e6b246f8167ee00d651efb86a5f5090bebb79f0269a64f09dbfafc166c0"} Sep 07 10:41:56.220570 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.220512 2565 scope.go:117] "RemoveContainer" containerID="37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74" Sep 07 10:41:56.230541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.230522 2565 scope.go:117] "RemoveContainer" containerID="978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa" Sep 07 10:41:56.239737 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.239719 2565 scope.go:117] "RemoveContainer" containerID="37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74" Sep 07 10:41:56.240043 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:41:56.240024 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74\": container with ID starting with 37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74 not found: ID does not exist" containerID="37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74" Sep 07 10:41:56.240121 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.240058 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74"} err="failed to get container status \"37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74\": rpc error: code = NotFound desc = could not find container \"37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74\": container with ID starting with 37f3161395a5bbffafd3518663362da09c46bd047ecf1e8a4f6fa8c25295ed74 not found: ID does not exist" Sep 07 10:41:56.240121 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.240084 2565 scope.go:117] "RemoveContainer" containerID="978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa" Sep 07 10:41:56.240391 ip-10-0-132-168 kubenswrapper[2565]: E0907 10:41:56.240374 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa\": container with ID starting with 978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa not found: ID does not exist" containerID="978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa" Sep 07 10:41:56.240436 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.240399 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa"} err="failed to get container status \"978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa\": rpc error: code = NotFound desc = could not find container \"978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa\": container with ID starting with 978fdf8267c56663f889e75586a45ba4872f181e88aa95a90ce14f501f9fa9fa not found: ID does not exist" Sep 07 10:41:56.245060 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.245011 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp"] Sep 07 10:41:56.253541 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.248225 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-93e41e-openshift-default-f5859f984-c2lnp"] Sep 07 10:41:56.544219 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:41:56.544189 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" path="/var/lib/kubelet/pods/1a918e9d-81f9-4f8f-9905-f4e91f738770/volumes" Sep 07 10:42:19.012469 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:42:19.012432 2565 scope.go:117] "RemoveContainer" containerID="719aada5c9631a5e00291e9eb8554c439b5dfabd0e6c4a559348f4a7077b815a" Sep 07 10:42:19.021700 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:42:19.021678 2565 scope.go:117] "RemoveContainer" containerID="0b16c20b80e77b89eb64573ebada83dde6b27ff715b9df33a77a3a765af9521c" Sep 07 10:42:19.029855 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:42:19.029836 2565 scope.go:117] "RemoveContainer" containerID="b50a481cd3631ee370808e3f830d8f82f91d7c6c786313f2ef2ae6356d377d5b" Sep 07 10:42:19.038207 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:42:19.038181 2565 scope.go:117] "RemoveContainer" containerID="debfff4f6df9d1ec48f18fd53cb1e6db3249e2872158bbf0fdafe9e2c2141174" Sep 07 10:43:19.081650 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:43:19.081618 2565 scope.go:117] "RemoveContainer" containerID="5b2911c6e18135db168226022bd731120913098d083f613ca0ade842f66c2fd1" Sep 07 10:43:19.090443 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:43:19.090424 2565 scope.go:117] "RemoveContainer" containerID="e2c5f1539bf63466c30bb571f33b9470bb55cc54be4af7454eb0c0905f9b3b48" Sep 07 10:46:18.715446 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:46:18.715414 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:46:18.718180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:46:18.718140 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:46:19.132208 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:46:19.132177 2565 scope.go:117] "RemoveContainer" containerID="a97fbf9b227dc33df10d850a3172cef0e28aee125f447fb2499bfbd83c3c02ba" Sep 07 10:46:19.143057 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:46:19.143030 2565 scope.go:117] "RemoveContainer" containerID="9d99e14695d4f6fbaf31ff4f83ddc248015470cd465f9602256489c26d861835" Sep 07 10:47:19.167001 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:47:19.166923 2565 scope.go:117] "RemoveContainer" containerID="947df8ca2f9f0f84d28481e72ac55020cd022a7a39bbc5c2d20707f05b8c4a88" Sep 07 10:47:19.176547 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:47:19.176524 2565 scope.go:117] "RemoveContainer" containerID="97d0a2a202f24420c15ce2a96737e1bd190e79957b5987ebfc7435a7799555ac" Sep 07 10:48:04.680124 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:04.680088 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zhpzs_07a4616c-f300-418e-8027-6b8efc43c741/manager/0.log" Sep 07 10:48:05.033662 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:05.033631 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-krp6k_9266e4c7-c4e2-467b-aa93-70fc1b738732/copy-manifests/0.log" Sep 07 10:48:05.051152 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:05.051122 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-krp6k_9266e4c7-c4e2-467b-aa93-70fc1b738732/manager/0.log" Sep 07 10:48:07.669957 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:07.669928 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv_401435cb-b468-417d-b7b5-d98787fa98f0/istio-proxy/0.log" Sep 07 10:48:07.789785 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:07.789758 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-x7qjh_54653ac5-5240-4e23-b341-8d14a8890ca6/discovery/0.log" Sep 07 10:48:08.148912 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:08.148878 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-94w8f_114f0b81-1d6c-45c7-a59f-0d5d210c49b2/istio-proxy/1.log" Sep 07 10:48:08.387600 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:08.387563 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jxjws_170e144a-1758-4aa1-9f7f-9130f65ce97a/payload-processing/0.log" Sep 07 10:48:08.524314 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:08.524283 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bd85f49dd-l2v9q_be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0/router/0.log" Sep 07 10:48:08.883743 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:08.883661 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jsqq7_a008c9dc-6f53-4f35-9cab-10cb26583986/main/0.log" Sep 07 10:48:08.999928 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:08.999893 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-b59c2_9840e152-cec6-442a-ba65-3c10d6bfe128/main/0.log" Sep 07 10:48:15.557518 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.557481 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhr67/must-gather-fqbcz"] Sep 07 10:48:15.558076 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558056 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" Sep 07 10:48:15.558180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558080 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" Sep 07 10:48:15.558180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558095 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" Sep 07 10:48:15.558180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558103 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" Sep 07 10:48:15.558180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558113 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" Sep 07 10:48:15.558180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558123 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a47ee8-6be1-4b0e-93f5-fa4fda879d5e" containerName="istio-proxy" Sep 07 10:48:15.558180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558138 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" Sep 07 10:48:15.558180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558148 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" Sep 07 10:48:15.558180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558181 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558190 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558232 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0211dc27-21c6-418e-afe7-419ae477594a" containerName="revoke-keys" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558241 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0211dc27-21c6-418e-afe7-419ae477594a" containerName="revoke-keys" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558365 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558380 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558389 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558399 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558407 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="964d80da-5b56-42ac-b508-b01a6e94e4a9" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558417 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0211dc27-21c6-418e-afe7-419ae477594a" containerName="revoke-keys" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558533 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558563 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558578 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" Sep 07 10:48:15.558693 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558586 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a918e9d-81f9-4f8f-9905-f4e91f738770" containerName="istio-proxy" Sep 07 10:48:15.559340 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.558705 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b52e926-8672-4aa0-9402-2492fd59dc88" containerName="istio-proxy" Sep 07 10:48:15.562098 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.562076 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhr67/must-gather-fqbcz" Sep 07 10:48:15.564678 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.564655 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhr67\"/\"kube-root-ca.crt\"" Sep 07 10:48:15.565631 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.565616 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhr67\"/\"openshift-service-ca.crt\"" Sep 07 10:48:15.565710 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.565637 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhr67\"/\"default-dockercfg-ftpfk\"" Sep 07 10:48:15.573217 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.573190 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhr67/must-gather-fqbcz"] Sep 07 10:48:15.593813 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.593770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5582ca97-db11-4f0c-860a-100ddd708cab-must-gather-output\") pod \"must-gather-fqbcz\" (UID: \"5582ca97-db11-4f0c-860a-100ddd708cab\") " pod="openshift-must-gather-dhr67/must-gather-fqbcz" Sep 07 10:48:15.593983 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.593931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbfs\" (UniqueName: \"kubernetes.io/projected/5582ca97-db11-4f0c-860a-100ddd708cab-kube-api-access-5zbfs\") pod \"must-gather-fqbcz\" (UID: \"5582ca97-db11-4f0c-860a-100ddd708cab\") " pod="openshift-must-gather-dhr67/must-gather-fqbcz" Sep 07 10:48:15.695180 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.695127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbfs\" (UniqueName: \"kubernetes.io/projected/5582ca97-db11-4f0c-860a-100ddd708cab-kube-api-access-5zbfs\") pod \"must-gather-fqbcz\" (UID: \"5582ca97-db11-4f0c-860a-100ddd708cab\") " pod="openshift-must-gather-dhr67/must-gather-fqbcz" Sep 07 10:48:15.695366 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.695225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5582ca97-db11-4f0c-860a-100ddd708cab-must-gather-output\") pod \"must-gather-fqbcz\" (UID: \"5582ca97-db11-4f0c-860a-100ddd708cab\") " pod="openshift-must-gather-dhr67/must-gather-fqbcz" Sep 07 10:48:15.695523 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.695501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5582ca97-db11-4f0c-860a-100ddd708cab-must-gather-output\") pod \"must-gather-fqbcz\" (UID: \"5582ca97-db11-4f0c-860a-100ddd708cab\") " pod="openshift-must-gather-dhr67/must-gather-fqbcz" Sep 07 10:48:15.703735 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.703718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbfs\" (UniqueName: \"kubernetes.io/projected/5582ca97-db11-4f0c-860a-100ddd708cab-kube-api-access-5zbfs\") pod \"must-gather-fqbcz\" (UID: \"5582ca97-db11-4f0c-860a-100ddd708cab\") " pod="openshift-must-gather-dhr67/must-gather-fqbcz" Sep 07 10:48:15.872591 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:15.872501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhr67/must-gather-fqbcz" Sep 07 10:48:16.145912 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:16.145888 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhr67/must-gather-fqbcz"] Sep 07 10:48:16.147910 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:48:16.147881 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5582ca97_db11_4f0c_860a_100ddd708cab.slice/crio-bc784ee45a7f158f1163bdfd744c0c6bd77635f147cb97af81af6b4f5cd18c5a WatchSource:0}: Error finding container bc784ee45a7f158f1163bdfd744c0c6bd77635f147cb97af81af6b4f5cd18c5a: Status 404 returned error can't find the container with id bc784ee45a7f158f1163bdfd744c0c6bd77635f147cb97af81af6b4f5cd18c5a Sep 07 10:48:16.149806 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:16.149791 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:48:16.218724 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:16.218692 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/must-gather-fqbcz" event={"ID":"5582ca97-db11-4f0c-860a-100ddd708cab","Type":"ContainerStarted","Data":"bc784ee45a7f158f1163bdfd744c0c6bd77635f147cb97af81af6b4f5cd18c5a"} Sep 07 10:48:17.226232 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:17.226122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/must-gather-fqbcz" event={"ID":"5582ca97-db11-4f0c-860a-100ddd708cab","Type":"ContainerStarted","Data":"a7a66d13590c0e6bda5035c7892beb7a163c727c52c7f9ed41bbcdf19602f82c"} Sep 07 10:48:17.226232 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:17.226190 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/must-gather-fqbcz" event={"ID":"5582ca97-db11-4f0c-860a-100ddd708cab","Type":"ContainerStarted","Data":"5ea337407bcb028265468cc8cd0a33b9bf00c92558b61de95da08f51dec070bc"} Sep 07 10:48:17.241397 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:17.241344 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhr67/must-gather-fqbcz" podStartSLOduration=1.415830691 podStartE2EDuration="2.241325796s" totalImagesPullingTime="825.495105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:17.240311703 +0000 UTC m=+2219.226592039" watchObservedRunningTime="2026-09-07 10:48:17.241325796 +0000 UTC m=+2219.227606131" Sep 07 10:48:18.499823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:18.499767 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f4j6j_0db793e3-82ec-40b8-ab3e-eddc545c4973/global-pull-secret-syncer/0.log" Sep 07 10:48:18.672964 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:18.672926 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vtq76_ca012ec4-4ff9-41e3-9135-3f7b477068d4/konnectivity-agent/0.log" Sep 07 10:48:18.752564 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:18.752479 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-168.ec2.internal_ea865860dc1ded4b85a7a2ca71f02386/haproxy/0.log" Sep 07 10:48:24.841988 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:24.841955 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7b35295b-4705-4447-b827-810ac109521c/alertmanager/0.log" Sep 07 10:48:24.866234 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:24.866203 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7b35295b-4705-4447-b827-810ac109521c/config-reloader/0.log" Sep 07 10:48:24.890947 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:24.890913 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7b35295b-4705-4447-b827-810ac109521c/kube-rbac-proxy-web/0.log" Sep 07 10:48:24.911602 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:24.911572 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7b35295b-4705-4447-b827-810ac109521c/kube-rbac-proxy/0.log" Sep 07 10:48:24.936886 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:24.936857 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7b35295b-4705-4447-b827-810ac109521c/kube-rbac-proxy-metric/0.log" Sep 07 10:48:24.957802 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:24.957769 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7b35295b-4705-4447-b827-810ac109521c/prom-label-proxy/0.log" Sep 07 10:48:24.986122 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:24.986091 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7b35295b-4705-4447-b827-810ac109521c/init-config-reloader/0.log" Sep 07 10:48:25.024842 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.024809 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-wzscw_6c6b7ec1-a764-41f0-8541-a7b52c6989a2/cluster-monitoring-operator/0.log" Sep 07 10:48:25.054862 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.054829 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mvt4r_3417365e-4283-4549-8f2f-b0c68d94931f/kube-state-metrics/0.log" Sep 07 10:48:25.075312 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.075286 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mvt4r_3417365e-4283-4549-8f2f-b0c68d94931f/kube-rbac-proxy-main/0.log" Sep 07 10:48:25.099567 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.099486 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mvt4r_3417365e-4283-4549-8f2f-b0c68d94931f/kube-rbac-proxy-self/0.log" Sep 07 10:48:25.151318 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.151291 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-fnvkc_46d5d069-14bb-4fbc-be12-2c85d05bd81b/monitoring-plugin/0.log" Sep 07 10:48:25.347825 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.347794 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xnqbp_a9674401-ea1d-4172-a89c-59f5089e6531/node-exporter/0.log" Sep 07 10:48:25.370848 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.370772 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xnqbp_a9674401-ea1d-4172-a89c-59f5089e6531/kube-rbac-proxy/0.log" Sep 07 10:48:25.393818 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.393788 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xnqbp_a9674401-ea1d-4172-a89c-59f5089e6531/init-textfile/0.log" Sep 07 10:48:25.730145 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.730113 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db95d656f-w9vx6_3aa69b0b-c80d-4170-899c-f81489f70610/telemeter-client/0.log" Sep 07 10:48:25.752985 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.752954 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db95d656f-w9vx6_3aa69b0b-c80d-4170-899c-f81489f70610/reload/0.log" Sep 07 10:48:25.773043 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:25.773017 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db95d656f-w9vx6_3aa69b0b-c80d-4170-899c-f81489f70610/kube-rbac-proxy/0.log" Sep 07 10:48:27.707401 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:27.707373 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-l922z_1d477d2b-4f2b-41ee-9800-73c0650da064/networking-console-plugin/0.log" Sep 07 10:48:28.348274 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:28.348233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/1.log" Sep 07 10:48:28.353205 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:28.353153 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nbkdc_1749d01c-371e-42dc-9c83-d31ecb1b2788/console-operator/2.log" Sep 07 10:48:28.905254 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:28.905220 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-tcmkq_2af8be4d-808e-4d17-b005-70c6f7717297/download-server/0.log" Sep 07 10:48:29.071585 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.071545 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j"] Sep 07 10:48:29.077119 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.077095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.092447 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.092416 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j"] Sep 07 10:48:29.147111 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.147059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-sys\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.147320 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.147129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-proc\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.147320 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.147181 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-lib-modules\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.147320 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.147211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-podres\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.147445 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.147330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2f2p\" (UniqueName: \"kubernetes.io/projected/d012b94f-9c5e-434a-9a87-75b02b2cd93c-kube-api-access-c2f2p\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.248728 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.248683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-sys\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.248924 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.248750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-proc\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.248924 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.248783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-lib-modules\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.248924 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.248810 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-podres\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.248924 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.248869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2f2p\" (UniqueName: \"kubernetes.io/projected/d012b94f-9c5e-434a-9a87-75b02b2cd93c-kube-api-access-c2f2p\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.249388 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.249363 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-sys\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.249492 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.249438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-proc\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.249567 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.249547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-lib-modules\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.249646 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.249629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d012b94f-9c5e-434a-9a87-75b02b2cd93c-podres\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.258926 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.258893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2f2p\" (UniqueName: \"kubernetes.io/projected/d012b94f-9c5e-434a-9a87-75b02b2cd93c-kube-api-access-c2f2p\") pod \"perf-node-gather-daemonset-ttg7j\" (UID: \"d012b94f-9c5e-434a-9a87-75b02b2cd93c\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.407342 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.407304 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:29.607610 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.607534 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-kvrrb_e187bc78-fd0c-4568-a569-788cf31a0982/volume-data-source-validator/0.log" Sep 07 10:48:29.704006 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:29.703832 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j"] Sep 07 10:48:29.709833 ip-10-0-132-168 kubenswrapper[2565]: W0907 10:48:29.709794 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd012b94f_9c5e_434a_9a87_75b02b2cd93c.slice/crio-7742d3679f5926cf32afbe3959a848206a4c8c466c00f93415730dad4ee09478 WatchSource:0}: Error finding container 7742d3679f5926cf32afbe3959a848206a4c8c466c00f93415730dad4ee09478: Status 404 returned error can't find the container with id 7742d3679f5926cf32afbe3959a848206a4c8c466c00f93415730dad4ee09478 Sep 07 10:48:30.353867 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:30.353828 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" event={"ID":"d012b94f-9c5e-434a-9a87-75b02b2cd93c","Type":"ContainerStarted","Data":"6fbca632c2fd127318d047d404228b2ef6f2c356a8c3136d88c0ac7cf5523ce4"} Sep 07 10:48:30.354390 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:30.353889 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" event={"ID":"d012b94f-9c5e-434a-9a87-75b02b2cd93c","Type":"ContainerStarted","Data":"7742d3679f5926cf32afbe3959a848206a4c8c466c00f93415730dad4ee09478"} Sep 07 10:48:30.354390 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:30.353942 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:30.372652 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:30.372583 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" podStartSLOduration=1.372559947 podStartE2EDuration="1.372559947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:30.370965849 +0000 UTC m=+2232.357246184" watchObservedRunningTime="2026-09-07 10:48:30.372559947 +0000 UTC m=+2232.358840283" Sep 07 10:48:30.782302 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:30.782270 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bfpfk_7a78daa9-fda7-4dbc-9465-17e251db07f6/dns/0.log" Sep 07 10:48:30.803198 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:30.803128 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bfpfk_7a78daa9-fda7-4dbc-9465-17e251db07f6/kube-rbac-proxy/0.log" Sep 07 10:48:30.935857 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:30.935816 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9qwl_154eb144-57c7-41b2-8ce3-d7efce4a447e/dns-node-resolver/0.log" Sep 07 10:48:31.495260 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:31.495210 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqhd9_ba852866-ea68-4579-8b0a-8f2c843d8023/node-ca/0.log" Sep 07 10:48:32.391449 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:32.391420 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv_401435cb-b468-417d-b7b5-d98787fa98f0/istio-proxy/0.log" Sep 07 10:48:32.977579 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:32.977549 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-x7qjh_54653ac5-5240-4e23-b341-8d14a8890ca6/discovery/0.log" Sep 07 10:48:33.059406 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:33.059369 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-94w8f_114f0b81-1d6c-45c7-a59f-0d5d210c49b2/istio-proxy/0.log" Sep 07 10:48:33.080579 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:33.080543 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-94w8f_114f0b81-1d6c-45c7-a59f-0d5d210c49b2/istio-proxy/1.log" Sep 07 10:48:33.176500 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:33.176461 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jxjws_170e144a-1758-4aa1-9f7f-9130f65ce97a/payload-processing/0.log" Sep 07 10:48:33.201944 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:33.201912 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bd85f49dd-l2v9q_be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0/router/0.log" Sep 07 10:48:34.103999 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:34.103960 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pbd8k_9f1d9389-f077-4289-92f0-19197641f90c/serve-healthcheck-canary/0.log" Sep 07 10:48:34.556285 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:34.556227 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-qkcbp_778d7208-c6a5-45c3-84d5-618bb54a7924/insights-operator/1.log" Sep 07 10:48:34.556847 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:34.556795 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-qkcbp_778d7208-c6a5-45c3-84d5-618bb54a7924/insights-operator/0.log" Sep 07 10:48:34.716347 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:34.716312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rs2dm_60e57c56-bb1b-42cd-a21c-90a1cae39c71/kube-rbac-proxy/0.log" Sep 07 10:48:34.737200 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:34.737153 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rs2dm_60e57c56-bb1b-42cd-a21c-90a1cae39c71/exporter/0.log" Sep 07 10:48:34.756662 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:34.756637 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rs2dm_60e57c56-bb1b-42cd-a21c-90a1cae39c71/extractor/0.log" Sep 07 10:48:36.371283 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:36.371256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-ttg7j" Sep 07 10:48:37.146823 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:37.146789 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zhpzs_07a4616c-f300-418e-8027-6b8efc43c741/manager/0.log" Sep 07 10:48:37.355812 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:37.355776 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-krp6k_9266e4c7-c4e2-467b-aa93-70fc1b738732/manager/0.log" Sep 07 10:48:37.370479 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:37.370454 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-krp6k_9266e4c7-c4e2-467b-aa93-70fc1b738732/copy-manifests/0.log" Sep 07 10:48:38.818574 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:38.818499 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c54b9f868-s8nvj_d481477b-c29f-45de-9658-825a4fcc5676/manager/0.log" Sep 07 10:48:39.558265 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:39.558226 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv_401435cb-b468-417d-b7b5-d98787fa98f0/istio-proxy/0.log" Sep 07 10:48:40.159182 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:40.156512 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-x7qjh_54653ac5-5240-4e23-b341-8d14a8890ca6/discovery/0.log" Sep 07 10:48:40.205278 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:40.205245 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-94w8f_114f0b81-1d6c-45c7-a59f-0d5d210c49b2/istio-proxy/0.log" Sep 07 10:48:40.225002 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:40.224965 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-94w8f_114f0b81-1d6c-45c7-a59f-0d5d210c49b2/istio-proxy/1.log" Sep 07 10:48:40.294381 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:40.294344 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jxjws_170e144a-1758-4aa1-9f7f-9130f65ce97a/payload-processing/0.log" Sep 07 10:48:40.309574 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:40.309546 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bd85f49dd-l2v9q_be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0/router/0.log" Sep 07 10:48:41.854453 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:41.854407 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cf8mnv_401435cb-b468-417d-b7b5-d98787fa98f0/istio-proxy/0.log" Sep 07 10:48:42.400959 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:42.400927 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-x7qjh_54653ac5-5240-4e23-b341-8d14a8890ca6/discovery/0.log" Sep 07 10:48:42.461591 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:42.461552 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-94w8f_114f0b81-1d6c-45c7-a59f-0d5d210c49b2/istio-proxy/0.log" Sep 07 10:48:42.482458 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:42.482428 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-94w8f_114f0b81-1d6c-45c7-a59f-0d5d210c49b2/istio-proxy/1.log" Sep 07 10:48:42.547492 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:42.547455 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jxjws_170e144a-1758-4aa1-9f7f-9130f65ce97a/payload-processing/0.log" Sep 07 10:48:42.567700 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:42.567656 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bd85f49dd-l2v9q_be1f4ce2-9f6c-4b8e-8d87-2f7250b78ad0/router/0.log" Sep 07 10:48:44.751076 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:44.751036 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-8c94h_be97b92a-7c7c-4cad-a5be-07bd910af233/kube-storage-version-migrator-operator/1.log" Sep 07 10:48:44.753392 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:44.753364 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-8c94h_be97b92a-7c7c-4cad-a5be-07bd910af233/kube-storage-version-migrator-operator/0.log" Sep 07 10:48:45.820362 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:45.820338 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c5vj_1f7afb78-4695-4bf1-9835-117132b6b761/kube-multus-additional-cni-plugins/0.log" Sep 07 10:48:45.842895 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:45.842861 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c5vj_1f7afb78-4695-4bf1-9835-117132b6b761/egress-router-binary-copy/0.log" Sep 07 10:48:45.875266 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:45.875230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c5vj_1f7afb78-4695-4bf1-9835-117132b6b761/cni-plugins/0.log" Sep 07 10:48:45.894506 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:45.894453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c5vj_1f7afb78-4695-4bf1-9835-117132b6b761/bond-cni-plugin/0.log" Sep 07 10:48:45.914503 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:45.914476 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c5vj_1f7afb78-4695-4bf1-9835-117132b6b761/routeoverride-cni/0.log" Sep 07 10:48:45.933768 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:45.933724 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c5vj_1f7afb78-4695-4bf1-9835-117132b6b761/whereabouts-cni-bincopy/0.log" Sep 07 10:48:45.953211 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:45.953181 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c5vj_1f7afb78-4695-4bf1-9835-117132b6b761/whereabouts-cni/0.log" Sep 07 10:48:46.333953 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:46.333798 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n7z4k_8eb29491-0135-47fe-9783-122b76372ef9/kube-multus/0.log" Sep 07 10:48:46.530055 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:46.529964 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lb6k7_05166e4c-db23-4e20-b194-d8adbf6ce954/network-metrics-daemon/0.log" Sep 07 10:48:46.546012 ip-10-0-132-168 kubenswrapper[2565]: I0907 10:48:46.545985 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lb6k7_05166e4c-db23-4e20-b194-d8adbf6ce954/kube-rbac-proxy/0.log"