Sep 18 21:03:54.697813 ip-10-0-132-187 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 18 21:03:54.697825 ip-10-0-132-187 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 18 21:03:54.697835 ip-10-0-132-187 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 18 21:03:54.698053 ip-10-0-132-187 systemd[1]: Failed to start Kubernetes Kubelet. Sep 18 21:04:04.857065 ip-10-0-132-187 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 18 21:04:04.857076 ip-10-0-132-187 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4955072f09af4cab8527dba4e0ef8fb4 -- Sep 18 21:06:09.544317 ip-10-0-132-187 systemd[1]: Starting Kubernetes Kubelet... Sep 18 21:06:13.102564 ip-10-0-132-187 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 21:06:13.102564 ip-10-0-132-187 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 21:06:13.102564 ip-10-0-132-187 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 21:06:13.102564 ip-10-0-132-187 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 21:06:13.102564 ip-10-0-132-187 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 21:06:13.103492 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.102667 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 21:06:13.107892 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107866 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 21:06:13.107892 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107883 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 21:06:13.107892 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107888 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 21:06:13.107892 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107891 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 21:06:13.107892 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107896 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 21:06:13.107892 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107899 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107903 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107906 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107909 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107912 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107915 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107918 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107921 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107924 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107927 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107929 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107932 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107935 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107938 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107941 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107943 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107946 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107949 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107967 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107970 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 21:06:13.108140 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107973 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107977 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107983 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107986 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107989 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107992 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107995 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.107998 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108001 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108004 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108006 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108009 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108012 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108015 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108018 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108022 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108025 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108027 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108030 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108033 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 21:06:13.108633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108036 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108039 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108042 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108045 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108047 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108050 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108053 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108056 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108059 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108062 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108065 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108067 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108074 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108079 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108082 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108085 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108088 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108091 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108094 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108097 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 21:06:13.109146 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108099 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108102 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108107 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108112 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108115 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108118 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108121 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108124 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108127 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108130 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108133 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108136 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108139 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108142 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108147 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108153 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108156 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108159 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108164 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 21:06:13.109644 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108167 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108170 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108173 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108176 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108179 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108182 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108185 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108188 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108191 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108194 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108197 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108199 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108203 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108205 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108208 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108943 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108950 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108969 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108974 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108979 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 21:06:13.110126 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108982 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108986 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108989 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108992 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108994 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.108997 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109000 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109003 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109006 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109008 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109011 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109014 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109017 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109020 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109023 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109026 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109029 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109033 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109036 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109038 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 21:06:13.110609 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109041 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109044 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109046 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109049 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109052 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109054 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109057 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109060 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109063 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109066 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109068 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109071 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109074 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109077 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109080 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109083 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109086 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109088 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109091 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109094 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 21:06:13.111117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109097 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109099 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109102 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109105 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109108 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109111 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109116 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109118 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109121 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109124 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109127 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109130 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109133 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109135 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109140 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109144 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109148 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109152 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109155 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 21:06:13.111601 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109157 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109160 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109163 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109166 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109169 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109171 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109174 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109177 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109180 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109183 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109186 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109188 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109191 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109194 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109198 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109201 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109203 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109206 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109208 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109212 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 21:06:13.112080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109214 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109217 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109220 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109223 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109226 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109228 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109231 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109234 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109236 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109239 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109242 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109245 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109248 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109251 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109253 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109657 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109663 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109666 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109669 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 21:06:13.112593 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109672 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109674 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109677 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109680 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109683 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109686 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109688 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109691 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109694 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109697 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109699 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109702 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109705 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109708 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109711 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109714 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109717 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109720 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109723 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109726 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 21:06:13.113071 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109728 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109731 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109734 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109737 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109739 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109744 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109748 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109751 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109754 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109758 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109762 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109765 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109767 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109770 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109773 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109776 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109779 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109781 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109784 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 21:06:13.113560 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109787 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109790 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109793 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109795 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109799 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109801 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109806 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109809 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109812 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109814 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109817 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109821 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109825 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109827 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109830 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109833 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109836 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109839 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109841 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 21:06:13.114095 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109844 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109847 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109850 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109853 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109855 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109858 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109861 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109864 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109867 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109870 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109873 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109875 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109878 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109881 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109884 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109886 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109889 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109892 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109894 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109897 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 21:06:13.114568 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109903 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109906 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109908 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109911 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109914 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109916 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109919 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109922 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109927 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109929 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109932 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109935 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109937 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109940 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109943 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109945 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.109948 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110037 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110072 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110078 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110083 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 21:06:13.115081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110092 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110095 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110099 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110103 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110107 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110110 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110114 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110117 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110120 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110123 2577 flags.go:64] FLAG: --cgroup-root="" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110126 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110131 2577 flags.go:64] FLAG: --client-ca-file="" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110135 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110139 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110143 2577 flags.go:64] FLAG: --cluster-domain="" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110146 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110150 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110153 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110157 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110162 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110165 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110169 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110172 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110175 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 18 21:06:13.115632 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110179 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110182 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110185 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110188 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110192 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110196 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110199 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110202 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110205 2577 flags.go:64] FLAG: --enable-server="true" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110208 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110212 2577 flags.go:64] FLAG: --event-burst="100" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110216 2577 flags.go:64] FLAG: --event-qps="50" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110219 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110222 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110225 2577 flags.go:64] FLAG: --eviction-hard="" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110229 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110232 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110235 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110238 2577 flags.go:64] FLAG: --eviction-soft="" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110242 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110246 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110250 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110253 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110256 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110259 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 21:06:13.116240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110262 2577 flags.go:64] FLAG: --feature-gates="" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110266 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110271 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110274 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110277 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110280 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110283 2577 flags.go:64] FLAG: --help="false" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110287 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110290 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110293 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110296 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110300 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110303 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110306 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110309 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110312 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110315 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110319 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110322 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110325 2577 flags.go:64] FLAG: --kube-reserved="" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110328 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110331 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110335 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110338 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 21:06:13.116866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110341 2577 flags.go:64] FLAG: --lock-file="" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110344 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110348 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110351 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110356 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110360 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110363 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110366 2577 flags.go:64] FLAG: --logging-format="text" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110369 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110373 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110377 2577 flags.go:64] FLAG: --manifest-url="" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110380 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110385 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110388 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110392 2577 flags.go:64] FLAG: --max-pods="110" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110395 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110398 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110401 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110404 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110408 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110411 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110414 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110421 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110425 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110428 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 21:06:13.117465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110431 2577 flags.go:64] FLAG: --pod-cidr="" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110434 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110439 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110442 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110445 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110448 2577 flags.go:64] FLAG: --port="10250" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110451 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110454 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ea3721e3d50abb11" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110458 2577 flags.go:64] FLAG: --qos-reserved="" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110461 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110466 2577 flags.go:64] FLAG: --register-node="true" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110469 2577 flags.go:64] FLAG: --register-with-taints="" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110472 2577 flags.go:64] FLAG: --registry-burst="10" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110476 2577 flags.go:64] FLAG: --registry-qps="5" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110479 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110482 2577 flags.go:64] FLAG: --reserved-memory="" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110486 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110491 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110494 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110497 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110500 2577 flags.go:64] FLAG: --runonce="false" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110503 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110506 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110509 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110512 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 21:06:13.118103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110516 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110519 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110522 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110525 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110528 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110531 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110534 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110537 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110541 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110544 2577 flags.go:64] FLAG: --system-cgroups="" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110547 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110552 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110555 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110558 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110563 2577 flags.go:64] FLAG: --tls-min-version="" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110566 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110570 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110575 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110578 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110581 2577 flags.go:64] FLAG: --v="2" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110585 2577 flags.go:64] FLAG: --version="false" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110590 2577 flags.go:64] FLAG: --vmodule="" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110594 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.110598 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110733 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 21:06:13.118726 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110738 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110741 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110744 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110747 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110750 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110753 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110756 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110759 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110762 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110765 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110768 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110771 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110774 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110777 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110781 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110785 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110788 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110791 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110795 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 21:06:13.119344 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110800 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110804 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110808 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110813 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110816 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110819 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110822 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110825 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110828 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110831 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110834 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110837 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110841 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110844 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110847 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110850 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110853 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110855 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110858 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110861 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 21:06:13.119831 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110864 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110866 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110869 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110872 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110875 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110877 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110880 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110883 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110886 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110889 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110891 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110894 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110896 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110899 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110902 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110904 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110907 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110910 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110912 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110915 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 21:06:13.120413 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110918 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110921 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110923 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110927 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110931 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110934 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110937 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110939 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110942 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110945 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110948 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110950 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110965 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110968 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110972 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110977 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110980 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110983 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110985 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110988 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 21:06:13.120904 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110991 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110993 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110996 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.110999 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111001 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111004 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111007 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111010 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111012 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111015 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111018 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111021 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111023 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111026 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111030 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111034 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111039 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111042 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 21:06:13.121423 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.111045 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.111525 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.118410 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.118428 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.118448 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.118457 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118520 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118525 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118530 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118535 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118538 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118541 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118544 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118547 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 21:06:13.121939 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118550 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118553 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118556 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118559 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118562 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118565 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118568 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118571 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118574 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118576 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118579 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118582 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118585 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118588 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118591 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118595 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118598 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118601 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118604 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 21:06:13.122310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118607 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118610 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118615 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118618 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118621 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118624 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118627 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118630 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118633 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118635 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118638 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118641 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118644 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118646 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118649 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118652 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118655 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118657 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118661 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 21:06:13.122851 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118665 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118669 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118672 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118675 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118678 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118681 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118684 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118686 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118689 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118692 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118695 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118698 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118701 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118703 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118706 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118709 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118713 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118716 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118719 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118722 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 21:06:13.123526 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118724 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118727 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118730 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118733 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118736 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118738 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118741 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118744 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118747 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118750 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118753 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118755 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118758 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118761 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118764 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118766 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118769 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118772 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118775 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118778 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 21:06:13.124389 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118780 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118783 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118786 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118788 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118791 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118794 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118796 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118800 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118803 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118806 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118809 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118812 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118814 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.118819 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118976 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 21:06:13.124998 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118983 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118987 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118990 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118993 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118996 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.118999 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119003 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119006 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119009 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119011 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119014 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119017 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119020 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119023 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119025 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119030 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119033 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119036 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119040 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119042 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 21:06:13.125456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119045 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119048 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119051 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119053 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119057 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119060 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119063 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119073 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119076 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119079 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119082 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119084 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119087 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119090 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119092 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119095 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119098 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119100 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119103 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119106 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 21:06:13.126050 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119108 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119111 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119114 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119116 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119119 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119122 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119125 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119127 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119130 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119132 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119135 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119138 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119140 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119143 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119146 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119148 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119151 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119154 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119157 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119160 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 21:06:13.126581 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119168 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119171 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119174 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119176 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119179 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119181 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119184 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119187 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119189 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119192 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119194 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119197 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119200 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119203 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119205 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119208 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119211 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119214 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119216 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119219 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 21:06:13.127377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119222 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119224 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119227 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119230 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119233 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119235 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119238 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119241 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119243 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119247 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119250 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119254 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119258 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119267 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119270 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119273 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119276 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 21:06:13.127869 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:13.119279 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 21:06:13.128384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.119284 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 21:06:13.128384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.119417 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 21:06:13.128384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.121270 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 21:06:13.128384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.122236 2577 server.go:1019] "Starting client certificate rotation" Sep 18 21:06:13.128384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.122356 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 21:06:13.128384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.123096 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 21:06:13.128384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.124813 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 21:06:13.128384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.126121 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 21:06:13.139709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.139686 2577 log.go:25] "Validated CRI v1 runtime API" Sep 18 21:06:13.142613 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.142599 2577 log.go:25] "Validated CRI v1 image API" Sep 18 21:06:13.143547 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.143524 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 21:06:13.145881 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.145860 2577 fs.go:135] Filesystem UUIDs: map[16737ca5-b447-45d4-9c5d-ce82e05457f8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a883eda5-27cf-4bff-a299-4045f71cdc85:/dev/nvme0n1p3] Sep 18 21:06:13.145940 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.145881 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 21:06:13.151074 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.150948 2577 manager.go:216] Machine: {Timestamp:2026-09-18 21:06:13.149510539 +0000 UTC m=+2.079256405 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099566 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f2953741338e3650719ba2a1fea30 SystemUUID:ec2f2953-7413-38e3-6507-19ba2a1fea30 BootID:4955072f-09af-4cab-8527-dba4e0ef8fb4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:f1:29:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:f1:29:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:58:4e:40:77:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 21:06:13.151074 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.151066 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 21:06:13.151205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.151171 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 21:06:13.152576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.152553 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 21:06:13.152713 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.152579 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 21:06:13.153918 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.153906 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 21:06:13.153967 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.153921 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 21:06:13.153967 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.153929 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 21:06:13.153967 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.153940 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 21:06:13.154052 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.154016 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 21:06:13.155756 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.155740 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 21:06:13.157261 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157248 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 18 21:06:13.157374 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157364 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 21:06:13.157403 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157394 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 18 21:06:13.157474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157405 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 21:06:13.157474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157420 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 21:06:13.157474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157431 2577 kubelet.go:398] "Adding apiserver pod source" Sep 18 21:06:13.157474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157443 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 21:06:13.157883 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157869 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 21:06:13.157945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.157890 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 21:06:13.160115 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.160098 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 21:06:13.162330 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.162317 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 21:06:13.162379 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.162346 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 21:06:13.163427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163416 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 21:06:13.163466 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163430 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 21:06:13.163466 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163436 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 21:06:13.163466 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163442 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 21:06:13.163466 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163451 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 21:06:13.163466 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163456 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 21:06:13.163607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163470 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 21:06:13.163607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163476 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 21:06:13.163607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163482 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 21:06:13.163607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163487 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 21:06:13.163607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163495 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 21:06:13.163607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163500 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 21:06:13.163607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163522 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 21:06:13.163607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.163527 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 21:06:13.165874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.165853 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 21:06:13.165966 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.165876 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 21:06:13.166014 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.165965 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 21:06:13.167264 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.167251 2577 server.go:1268] "Started kubelet" Sep 18 21:06:13.167328 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.167306 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 21:06:13.167451 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.167399 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 21:06:13.167502 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.167473 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 21:06:13.167986 ip-10-0-132-187 systemd[1]: Started Kubernetes Kubelet. Sep 18 21:06:13.197562 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.197540 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 21:06:13.197667 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.197645 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qn5cf" Sep 18 21:06:13.197720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.197672 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qn5cf" Sep 18 21:06:13.199009 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.198539 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 18 21:06:13.200991 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.200970 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 21:06:13.203543 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.203527 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 21:06:13.203543 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.203535 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 21:06:13.203686 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.203632 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.204167 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204148 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 21:06:13.204167 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204150 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 21:06:13.204313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204179 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 21:06:13.204313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204273 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 18 21:06:13.204313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204280 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 18 21:06:13.204410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204353 2577 factory.go:55] Registering systemd factory Sep 18 21:06:13.204410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204368 2577 factory.go:223] Registration of the systemd container factory successfully Sep 18 21:06:13.204575 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204563 2577 factory.go:153] Registering CRI-O factory Sep 18 21:06:13.204575 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204575 2577 factory.go:223] Registration of the crio container factory successfully Sep 18 21:06:13.204659 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204621 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 21:06:13.204659 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204639 2577 factory.go:103] Registering Raw factory Sep 18 21:06:13.204659 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.204653 2577 manager.go:1196] Started watching for new ooms in manager Sep 18 21:06:13.205124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.205109 2577 manager.go:349] Starting recovery of all containers Sep 18 21:06:13.212380 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.212363 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 21:06:13.215205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.215186 2577 manager.go:354] Recovery completed Sep 18 21:06:13.215465 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.215445 2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-187.ec2.internal\" not found" node="ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.219586 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.219574 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 21:06:13.221067 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.221045 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientMemory" Sep 18 21:06:13.221127 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.221090 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 21:06:13.221127 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.221106 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientPID" Sep 18 21:06:13.221602 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.221590 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 21:06:13.221653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.221601 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 21:06:13.221653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.221624 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 18 21:06:13.221653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.221603 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 21:06:13.222301 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.222290 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientMemory" Sep 18 21:06:13.222350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.222312 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 21:06:13.222350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.222326 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientPID" Sep 18 21:06:13.222668 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.222657 2577 policy_none.go:49] "None policy: Start" Sep 18 21:06:13.222697 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.222672 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 21:06:13.222697 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.222682 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 21:06:13.223558 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.223547 2577 policy_none.go:47] "Start" Sep 18 21:06:13.262935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.262718 2577 manager.go:337] "Starting Device Plugin manager" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.263016 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.263039 2577 server.go:87] "Starting device plugin registration server" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.263246 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.263268 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.263363 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.263476 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.263485 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.263873 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 21:06:13.277007 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.263919 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.350743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.350706 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 21:06:13.351886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.351873 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 21:06:13.351993 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.351923 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 21:06:13.351993 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.351948 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 21:06:13.352096 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.351997 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 21:06:13.354395 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.354352 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 21:06:13.364296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.364279 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 21:06:13.365106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.365091 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientMemory" Sep 18 21:06:13.365180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.365120 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 21:06:13.365180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.365131 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientPID" Sep 18 21:06:13.365180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.365153 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.373934 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.373922 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.373993 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.373940 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-187.ec2.internal\": node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.391216 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.391200 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.452725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.452681 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal"] Sep 18 21:06:13.452821 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.452778 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 21:06:13.453944 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.453930 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientMemory" Sep 18 21:06:13.454020 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.453975 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 21:06:13.454020 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.453989 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientPID" Sep 18 21:06:13.455282 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.455269 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 21:06:13.455424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.455404 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.455461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.455437 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 21:06:13.456043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.456015 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientMemory" Sep 18 21:06:13.456043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.456032 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientMemory" Sep 18 21:06:13.456198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.456047 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 21:06:13.456198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.456063 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientPID" Sep 18 21:06:13.456198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.456050 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 21:06:13.456198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.456100 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientPID" Sep 18 21:06:13.457412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.457395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.457491 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.457427 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 21:06:13.458157 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.458142 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientMemory" Sep 18 21:06:13.458227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.458173 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 21:06:13.458227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.458187 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeHasSufficientPID" Sep 18 21:06:13.486592 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.486576 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-187.ec2.internal\" not found" node="ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.490787 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.490772 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-187.ec2.internal\" not found" node="ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.491762 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.491748 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.506482 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.506462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a2162d0f83354e882cf0ecc5729b614-config\") pod \"kube-apiserver-proxy-ip-10-0-132-187.ec2.internal\" (UID: \"8a2162d0f83354e882cf0ecc5729b614\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.506565 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.506485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1510db1b6cb9287988e62a0f9d001698-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal\" (UID: \"1510db1b6cb9287988e62a0f9d001698\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.506565 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.506505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1510db1b6cb9287988e62a0f9d001698-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal\" (UID: \"1510db1b6cb9287988e62a0f9d001698\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.592559 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.592540 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.607513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.607452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1510db1b6cb9287988e62a0f9d001698-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal\" (UID: \"1510db1b6cb9287988e62a0f9d001698\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.607513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.607475 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1510db1b6cb9287988e62a0f9d001698-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal\" (UID: \"1510db1b6cb9287988e62a0f9d001698\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.607513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.607492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1510db1b6cb9287988e62a0f9d001698-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal\" (UID: \"1510db1b6cb9287988e62a0f9d001698\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.607686 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.607519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a2162d0f83354e882cf0ecc5729b614-config\") pod \"kube-apiserver-proxy-ip-10-0-132-187.ec2.internal\" (UID: \"8a2162d0f83354e882cf0ecc5729b614\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.607686 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.607583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1510db1b6cb9287988e62a0f9d001698-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal\" (UID: \"1510db1b6cb9287988e62a0f9d001698\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.607686 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.607588 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a2162d0f83354e882cf0ecc5729b614-config\") pod \"kube-apiserver-proxy-ip-10-0-132-187.ec2.internal\" (UID: \"8a2162d0f83354e882cf0ecc5729b614\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.692902 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.692859 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.789422 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.789396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.793666 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.793513 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.794610 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:13.794599 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" Sep 18 21:06:13.894133 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.894063 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:13.994558 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:13.994535 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:14.095072 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:14.095022 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:14.122531 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.122512 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 21:06:14.122987 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.122629 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 21:06:14.122987 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.122676 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 21:06:14.195578 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:14.195512 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-187.ec2.internal\" not found" Sep 18 21:06:14.202672 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.202645 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 21:01:13 +0000 UTC" deadline="2028-03-18 13:14:16.832134371 +0000 UTC" Sep 18 21:06:14.202672 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.202669 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13120h8m2.629468521s" Sep 18 21:06:14.250883 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.250862 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 21:06:14.304545 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.304516 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" Sep 18 21:06:14.314458 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.314438 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 21:06:14.316095 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.316076 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" Sep 18 21:06:14.328182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.328165 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 21:06:14.360228 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:14.360192 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2162d0f83354e882cf0ecc5729b614.slice/crio-09ee0ef9897bc2a0cecb6c984c7174ac5ba2ee66b8fcc062a21b5e772e91eca2 WatchSource:0}: Error finding container 09ee0ef9897bc2a0cecb6c984c7174ac5ba2ee66b8fcc062a21b5e772e91eca2: Status 404 returned error can't find the container with id 09ee0ef9897bc2a0cecb6c984c7174ac5ba2ee66b8fcc062a21b5e772e91eca2 Sep 18 21:06:14.360814 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:14.360797 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1510db1b6cb9287988e62a0f9d001698.slice/crio-3588d683fe8fb078cdefaa99f705792a507774dcd78684222950c29898d93560 WatchSource:0}: Error finding container 3588d683fe8fb078cdefaa99f705792a507774dcd78684222950c29898d93560: Status 404 returned error can't find the container with id 3588d683fe8fb078cdefaa99f705792a507774dcd78684222950c29898d93560 Sep 18 21:06:14.366705 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.366693 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 21:06:14.413757 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:14.413733 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 21:06:15.158215 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.158134 2577 apiserver.go:52] "Watching apiserver" Sep 18 21:06:15.166557 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.166528 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 21:06:15.167016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.166986 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal","openshift-multus/multus-additional-cni-plugins-js4j4","openshift-cluster-node-tuning-operator/tuned-gxlff","openshift-multus/multus-hptln","openshift-multus/network-metrics-daemon-rt75r","openshift-network-diagnostics/network-check-target-mb586","openshift-network-operator/iptables-alerter-cr7wf","openshift-ovn-kubernetes/ovnkube-node-nzszz","kube-system/global-pull-secret-syncer-6xgxk","kube-system/konnectivity-agent-55w59","kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4","openshift-dns/node-resolver-4588v","openshift-image-registry/node-ca-9zlx2"] Sep 18 21:06:15.168742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.168719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.169945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.169917 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.171693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.171044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.171693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.171175 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 21:06:15.171693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.171186 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 21:06:15.171693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.171383 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 21:06:15.171693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.171466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rbkpq\"" Sep 18 21:06:15.172615 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.172237 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 21:06:15.172615 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.172383 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mc7q5\"" Sep 18 21:06:15.172755 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.172657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hptln" Sep 18 21:06:15.173207 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.173087 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 21:06:15.173207 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.173091 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 21:06:15.173875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.173459 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 21:06:15.173875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.173500 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 21:06:15.173875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.173713 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jnwtj\"" Sep 18 21:06:15.174866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.174851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:15.174980 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.174920 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:15.176053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.176023 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:15.176157 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.176078 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:15.176227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.176174 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 21:06:15.176227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.176199 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 21:06:15.176343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.176326 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 21:06:15.176548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.176532 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w72wr\"" Sep 18 21:06:15.176612 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.176558 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 21:06:15.177688 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.177390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.177688 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.177457 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:15.178833 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.178812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.180314 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.180292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.180974 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.180723 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 21:06:15.180974 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.180773 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 21:06:15.181342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.181185 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c7cw4\"" Sep 18 21:06:15.181342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.181251 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 21:06:15.182356 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.182108 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:15.182356 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.182181 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 21:06:15.182511 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.182405 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 21:06:15.182511 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.182502 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4x6fl\"" Sep 18 21:06:15.184503 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.184464 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mkx42\"" Sep 18 21:06:15.184606 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.184554 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.184606 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.184561 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 21:06:15.184717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.184474 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 21:06:15.186029 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.186013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.186570 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.186549 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 21:06:15.186870 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.186840 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 21:06:15.187243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.187202 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xmk5m\"" Sep 18 21:06:15.187752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.187730 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 21:06:15.187831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.187752 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 21:06:15.187831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.187766 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 21:06:15.188397 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.188375 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 21:06:15.189703 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.188586 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fstlz\"" Sep 18 21:06:15.189801 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.189487 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 21:06:15.189801 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.189553 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 21:06:15.192075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.190056 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 21:06:15.203915 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.203889 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 21:06:15.205334 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.205305 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 21:06:15.216986 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.216964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.217077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.217077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24whj\" (UniqueName: \"kubernetes.io/projected/86bef309-7448-4e16-a947-c916df0fb672-kube-api-access-24whj\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.217077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysctl-conf\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.217077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6fc\" (UniqueName: \"kubernetes.io/projected/c61c453c-8a93-4f00-8856-74528a555127-kube-api-access-wf6fc\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.217077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c22257-3ecd-4264-89ef-560ecd2a6c41-host\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.217077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69f29fa6-0d0c-487a-ba87-8b99fbf8d364-konnectivity-ca\") pod \"konnectivity-agent-55w59\" (UID: \"69f29fa6-0d0c-487a-ba87-8b99fbf8d364\") " pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-registration-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79faf647-1593-440d-b79d-383738434d06-hosts-file\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29q4g\" (UniqueName: \"kubernetes.io/projected/79faf647-1593-440d-b79d-383738434d06-kube-api-access-29q4g\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf72364b-d85e-432b-b5b9-6ea60f72f68f-cni-binary-copy\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-kubelet\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-socket-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-node-log\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.217372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217361 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-ovnkube-script-lib\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69f29fa6-0d0c-487a-ba87-8b99fbf8d364-agent-certs\") pod \"konnectivity-agent-55w59\" (UID: \"69f29fa6-0d0c-487a-ba87-8b99fbf8d364\") " pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217473 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45gc\" (UniqueName: \"kubernetes.io/projected/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-kube-api-access-c45gc\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79faf647-1593-440d-b79d-383738434d06-tmp-dir\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-run-netns\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-cni-netd\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-socket-dir-parent\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-cni-multus\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-cni-bin\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-log-socket\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.217803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-env-overrides\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217814 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdwz\" (UniqueName: \"kubernetes.io/projected/34f89201-910a-440a-a3a1-e33ded838c8a-kube-api-access-bzdwz\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-device-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217858 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-cni-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217886 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-conf-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217907 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjpc9\" (UniqueName: \"kubernetes.io/projected/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-kube-api-access-wjpc9\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-cnibin\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217928 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217965 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-etc-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.217989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-modprobe-d\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-kubernetes\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-run\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-k8s-cni-cncf-io\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-hostroot\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-daemon-config\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.218326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-sys-fs\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-system-cni-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-multus-certs\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218221 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66d8cb46-3ba4-4b18-b769-a7667c20495c-dbus\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-systemd-units\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-lib-modules\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86bef309-7448-4e16-a947-c916df0fb672-ovn-node-metrics-cert\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-system-cni-dir\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-var-lib-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmp5m\" (UniqueName: \"kubernetes.io/projected/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-kube-api-access-tmp5m\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-sys\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61c453c-8a93-4f00-8856-74528a555127-tmp\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.219056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23c22257-3ecd-4264-89ef-560ecd2a6c41-serviceca\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-kubelet\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-ovnkube-config\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-iptables-alerter-script\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-var-lib-kubelet\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5246b\" (UniqueName: \"kubernetes.io/projected/23c22257-3ecd-4264-89ef-560ecd2a6c41-kube-api-access-5246b\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-cnibin\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-slash\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-host-slash\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-systemd\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-host\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-etc-kubernetes\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8jx\" (UniqueName: \"kubernetes.io/projected/bf72364b-d85e-432b-b5b9-6ea60f72f68f-kube-api-access-pm8jx\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66d8cb46-3ba4-4b18-b769-a7667c20495c-kubelet-config\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-cni-bin\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-systemd\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.218982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-ovn\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.219618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.219007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysctl-d\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.220218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.219026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61c453c-8a93-4f00-8856-74528a555127-etc-tuned\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.220218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.219064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-netns\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.220218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.219097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysconfig\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.247656 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.247637 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2ct75" Sep 18 21:06:15.255479 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.255455 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 21:06:15.256692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.256670 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2ct75" Sep 18 21:06:15.319670 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-kubelet\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.319670 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-ovnkube-config\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-iptables-alerter-script\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-var-lib-kubelet\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5246b\" (UniqueName: \"kubernetes.io/projected/23c22257-3ecd-4264-89ef-560ecd2a6c41-kube-api-access-5246b\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-cnibin\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319759 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-kubelet\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-slash\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-host-slash\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-systemd\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-host\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-etc-kubernetes\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8jx\" (UniqueName: \"kubernetes.io/projected/bf72364b-d85e-432b-b5b9-6ea60f72f68f-kube-api-access-pm8jx\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.319917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66d8cb46-3ba4-4b18-b769-a7667c20495c-kubelet-config\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-cni-bin\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-systemd\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.319993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-ovn\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysctl-d\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61c453c-8a93-4f00-8856-74528a555127-etc-tuned\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-netns\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-var-lib-kubelet\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysconfig\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320128 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-cnibin\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysconfig\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24whj\" (UniqueName: \"kubernetes.io/projected/86bef309-7448-4e16-a947-c916df0fb672-kube-api-access-24whj\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysctl-conf\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6fc\" (UniqueName: \"kubernetes.io/projected/c61c453c-8a93-4f00-8856-74528a555127-kube-api-access-wf6fc\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c22257-3ecd-4264-89ef-560ecd2a6c41-host\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.320507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69f29fa6-0d0c-487a-ba87-8b99fbf8d364-konnectivity-ca\") pod \"konnectivity-agent-55w59\" (UID: \"69f29fa6-0d0c-487a-ba87-8b99fbf8d364\") " pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-registration-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79faf647-1593-440d-b79d-383738434d06-hosts-file\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-iptables-alerter-script\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320451 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-host-slash\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-systemd\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320569 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-host\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320608 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-etc-kubernetes\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29q4g\" (UniqueName: \"kubernetes.io/projected/79faf647-1593-440d-b79d-383738434d06-kube-api-access-29q4g\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-ovnkube-config\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf72364b-d85e-432b-b5b9-6ea60f72f68f-cni-binary-copy\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-kubelet\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-slash\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-socket-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-node-log\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-ovnkube-script-lib\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-cni-bin\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69f29fa6-0d0c-487a-ba87-8b99fbf8d364-agent-certs\") pod \"konnectivity-agent-55w59\" (UID: \"69f29fa6-0d0c-487a-ba87-8b99fbf8d364\") " pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45gc\" (UniqueName: \"kubernetes.io/projected/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-kube-api-access-c45gc\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-ovn\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79faf647-1593-440d-b79d-383738434d06-tmp-dir\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-kubelet\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321120 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-socket-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321147 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-node-log\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf72364b-d85e-432b-b5b9-6ea60f72f68f-cni-binary-copy\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321248 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c22257-3ecd-4264-89ef-560ecd2a6c41-host\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321654 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79faf647-1593-440d-b79d-383738434d06-hosts-file\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-ovnkube-script-lib\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69f29fa6-0d0c-487a-ba87-8b99fbf8d364-konnectivity-ca\") pod \"konnectivity-agent-55w59\" (UID: \"69f29fa6-0d0c-487a-ba87-8b99fbf8d364\") " pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.320837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66d8cb46-3ba4-4b18-b769-a7667c20495c-kubelet-config\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321783 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysctl-d\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321821 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-registration-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-systemd\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.321923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321896 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-sysctl-conf\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.322720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.321929 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 21:06:15.322720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.322385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.323013 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.322984 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.323095 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323045 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-netns\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.323145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-run-netns\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323441 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-cni-netd\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323441 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79faf647-1593-440d-b79d-383738434d06-tmp-dir\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.323441 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-socket-dir-parent\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.323441 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323329 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-run-netns\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323504 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-cni-netd\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-cni-multus\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.323638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:15.323638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-cni-bin\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.323822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-log-socket\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323649 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-socket-dir-parent\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.323822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-run-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-cni-multus\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.323822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-log-socket\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.323822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323811 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-var-lib-cni-bin\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.324164 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.323881 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:15.324164 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.323919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-env-overrides\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.324164 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.323988 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs podName:9e35eaad-1b7d-40d2-a280-ef6d3c538a8c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:15.823934892 +0000 UTC m=+4.753680764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs") pod "network-metrics-daemon-rt75r" (UID: "9e35eaad-1b7d-40d2-a280-ef6d3c538a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:15.324164 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdwz\" (UniqueName: \"kubernetes.io/projected/34f89201-910a-440a-a3a1-e33ded838c8a-kube-api-access-bzdwz\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.324164 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-device-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.324164 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-cni-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.324164 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-conf-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.324470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjpc9\" (UniqueName: \"kubernetes.io/projected/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-kube-api-access-wjpc9\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:15.324470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-cnibin\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.324470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324341 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86bef309-7448-4e16-a947-c916df0fb672-env-overrides\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.324470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-etc-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.324470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324404 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.324470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-modprobe-d\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-kubernetes\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-run\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-k8s-cni-cncf-io\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-hostroot\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-daemon-config\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-sys-fs\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.324731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-system-cni-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-multus-certs\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66d8cb46-3ba4-4b18-b769-a7667c20495c-dbus\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-systemd-units\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-lib-modules\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-device-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86bef309-7448-4e16-a947-c916df0fb672-ovn-node-metrics-cert\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-system-cni-dir\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.324993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-var-lib-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325003 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-cni-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-conf-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325120 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-cnibin\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325150 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-systemd-units\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.325192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325200 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-modprobe-d\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-sys-fs\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-etc-kubernetes\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmp5m\" (UniqueName: \"kubernetes.io/projected/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-kube-api-access-tmp5m\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325331 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66d8cb46-3ba4-4b18-b769-a7667c20495c-dbus\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-lib-modules\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325378 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-k8s-cni-cncf-io\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-sys\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325419 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-system-cni-dir\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61c453c-8a93-4f00-8856-74528a555127-tmp\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-host-run-multus-certs\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.325467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23c22257-3ecd-4264-89ef-560ecd2a6c41-serviceca\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326050 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34f89201-910a-440a-a3a1-e33ded838c8a-system-cni-dir\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326128 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.327648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326169 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61c453c-8a93-4f00-8856-74528a555127-etc-tuned\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326211 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.326308 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.326390 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret podName:66d8cb46-3ba4-4b18-b769-a7667c20495c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:15.826367892 +0000 UTC m=+4.756113766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret") pod "global-pull-secret-syncer-6xgxk" (UID: "66d8cb46-3ba4-4b18-b769-a7667c20495c") : object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326408 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69f29fa6-0d0c-487a-ba87-8b99fbf8d364-agent-certs\") pod \"konnectivity-agent-55w59\" (UID: \"69f29fa6-0d0c-487a-ba87-8b99fbf8d364\") " pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-sys\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf72364b-d85e-432b-b5b9-6ea60f72f68f-hostroot\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-var-lib-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf72364b-d85e-432b-b5b9-6ea60f72f68f-multus-daemon-config\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.326994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34f89201-910a-440a-a3a1-e33ded838c8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.327048 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61c453c-8a93-4f00-8856-74528a555127-run\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.327063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86bef309-7448-4e16-a947-c916df0fb672-etc-openvswitch\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.328513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.327767 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23c22257-3ecd-4264-89ef-560ecd2a6c41-serviceca\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.329388 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.329369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86bef309-7448-4e16-a947-c916df0fb672-ovn-node-metrics-cert\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.329528 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.329444 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8jx\" (UniqueName: \"kubernetes.io/projected/bf72364b-d85e-432b-b5b9-6ea60f72f68f-kube-api-access-pm8jx\") pod \"multus-hptln\" (UID: \"bf72364b-d85e-432b-b5b9-6ea60f72f68f\") " pod="openshift-multus/multus-hptln" Sep 18 21:06:15.330132 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.330113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61c453c-8a93-4f00-8856-74528a555127-tmp\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.331097 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.331073 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5246b\" (UniqueName: \"kubernetes.io/projected/23c22257-3ecd-4264-89ef-560ecd2a6c41-kube-api-access-5246b\") pod \"node-ca-9zlx2\" (UID: \"23c22257-3ecd-4264-89ef-560ecd2a6c41\") " pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.331290 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.331272 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 21:06:15.331360 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.331297 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 21:06:15.331360 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.331311 2577 projected.go:196] Error preparing data for projected volume kube-api-access-h7nnk for pod openshift-network-diagnostics/network-check-target-mb586: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:15.331434 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.331370 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk podName:eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6 nodeName:}" failed. No retries permitted until 2026-09-18 21:06:15.831355692 +0000 UTC m=+4.761101563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h7nnk" (UniqueName: "kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk") pod "network-check-target-mb586" (UID: "eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:15.331484 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.331470 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29q4g\" (UniqueName: \"kubernetes.io/projected/79faf647-1593-440d-b79d-383738434d06-kube-api-access-29q4g\") pod \"node-resolver-4588v\" (UID: \"79faf647-1593-440d-b79d-383738434d06\") " pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.333406 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.333387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45gc\" (UniqueName: \"kubernetes.io/projected/7dbf713d-d757-4d4e-b94e-31ce8acaccd8-kube-api-access-c45gc\") pod \"aws-ebs-csi-driver-node-7wtg4\" (UID: \"7dbf713d-d757-4d4e-b94e-31ce8acaccd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.335649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.335627 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6fc\" (UniqueName: \"kubernetes.io/projected/c61c453c-8a93-4f00-8856-74528a555127-kube-api-access-wf6fc\") pod \"tuned-gxlff\" (UID: \"c61c453c-8a93-4f00-8856-74528a555127\") " pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.335903 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.335881 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24whj\" (UniqueName: \"kubernetes.io/projected/86bef309-7448-4e16-a947-c916df0fb672-kube-api-access-24whj\") pod \"ovnkube-node-nzszz\" (UID: \"86bef309-7448-4e16-a947-c916df0fb672\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.343411 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.343389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmp5m\" (UniqueName: \"kubernetes.io/projected/f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc-kube-api-access-tmp5m\") pod \"iptables-alerter-cr7wf\" (UID: \"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc\") " pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.343738 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.343716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdwz\" (UniqueName: \"kubernetes.io/projected/34f89201-910a-440a-a3a1-e33ded838c8a-kube-api-access-bzdwz\") pod \"multus-additional-cni-plugins-js4j4\" (UID: \"34f89201-910a-440a-a3a1-e33ded838c8a\") " pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.343801 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.343778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjpc9\" (UniqueName: \"kubernetes.io/projected/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-kube-api-access-wjpc9\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:15.356876 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.356832 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" event={"ID":"8a2162d0f83354e882cf0ecc5729b614","Type":"ContainerStarted","Data":"09ee0ef9897bc2a0cecb6c984c7174ac5ba2ee66b8fcc062a21b5e772e91eca2"} Sep 18 21:06:15.358513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.358485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" event={"ID":"1510db1b6cb9287988e62a0f9d001698","Type":"ContainerStarted","Data":"3588d683fe8fb078cdefaa99f705792a507774dcd78684222950c29898d93560"} Sep 18 21:06:15.484235 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.484152 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cr7wf" Sep 18 21:06:15.491348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.491328 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4588v" Sep 18 21:06:15.502108 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.502082 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gxlff" Sep 18 21:06:15.507800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.507777 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hptln" Sep 18 21:06:15.513372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.513354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9zlx2" Sep 18 21:06:15.520013 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.519989 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-js4j4" Sep 18 21:06:15.526509 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.526486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:15.533074 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.533050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:15.541834 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.541813 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" Sep 18 21:06:15.828799 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.828766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:15.828969 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.828820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:15.828969 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.828911 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:15.828969 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.828913 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:15.829075 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.828977 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs podName:9e35eaad-1b7d-40d2-a280-ef6d3c538a8c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:16.828949002 +0000 UTC m=+5.758694853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs") pod "network-metrics-daemon-rt75r" (UID: "9e35eaad-1b7d-40d2-a280-ef6d3c538a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:15.829075 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.828991 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret podName:66d8cb46-3ba4-4b18-b769-a7667c20495c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:16.82898486 +0000 UTC m=+5.758730710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret") pod "global-pull-secret-syncer-6xgxk" (UID: "66d8cb46-3ba4-4b18-b769-a7667c20495c") : object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:15.929731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:15.929694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:15.929888 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.929857 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 21:06:15.929888 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.929880 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 21:06:15.929998 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.929890 2577 projected.go:196] Error preparing data for projected volume kube-api-access-h7nnk for pod openshift-network-diagnostics/network-check-target-mb586: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:15.929998 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:15.929969 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk podName:eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6 nodeName:}" failed. No retries permitted until 2026-09-18 21:06:16.929936353 +0000 UTC m=+5.859682220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7nnk" (UniqueName: "kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk") pod "network-check-target-mb586" (UID: "eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:16.198883 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.198752 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61c453c_8a93_4f00_8856_74528a555127.slice/crio-99581a015b3031939ae6200363dc62cbe6f8bd9c05a00872bf9f10b260c60caf WatchSource:0}: Error finding container 99581a015b3031939ae6200363dc62cbe6f8bd9c05a00872bf9f10b260c60caf: Status 404 returned error can't find the container with id 99581a015b3031939ae6200363dc62cbe6f8bd9c05a00872bf9f10b260c60caf Sep 18 21:06:16.202350 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.202318 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f89201_910a_440a_a3a1_e33ded838c8a.slice/crio-b757cb1037e532f8e4e3b5681d728633ae471cb6846f3296ffa30a00038a5504 WatchSource:0}: Error finding container b757cb1037e532f8e4e3b5681d728633ae471cb6846f3296ffa30a00038a5504: Status 404 returned error can't find the container with id b757cb1037e532f8e4e3b5681d728633ae471cb6846f3296ffa30a00038a5504 Sep 18 21:06:16.205459 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.205428 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f29fa6_0d0c_487a_ba87_8b99fbf8d364.slice/crio-a3d1ea65f498a755158e4a375181de5a28ac7bbf94b92b1e1ff523a771e5cf3b WatchSource:0}: Error finding container a3d1ea65f498a755158e4a375181de5a28ac7bbf94b92b1e1ff523a771e5cf3b: Status 404 returned error can't find the container with id a3d1ea65f498a755158e4a375181de5a28ac7bbf94b92b1e1ff523a771e5cf3b Sep 18 21:06:16.206056 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.206027 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dbdfbd_1dfb_4187_9a9b_fb67b32baabc.slice/crio-e0337653dc224ccb21c355b74a94ef5e5e78abba7c717329142f94d25ccf0410 WatchSource:0}: Error finding container e0337653dc224ccb21c355b74a94ef5e5e78abba7c717329142f94d25ccf0410: Status 404 returned error can't find the container with id e0337653dc224ccb21c355b74a94ef5e5e78abba7c717329142f94d25ccf0410 Sep 18 21:06:16.207767 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.207719 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf72364b_d85e_432b_b5b9_6ea60f72f68f.slice/crio-4026c2012ed9c3d1fdf59a1083dffcc565cba2a86014afaafffbeb4b5cf38d46 WatchSource:0}: Error finding container 4026c2012ed9c3d1fdf59a1083dffcc565cba2a86014afaafffbeb4b5cf38d46: Status 404 returned error can't find the container with id 4026c2012ed9c3d1fdf59a1083dffcc565cba2a86014afaafffbeb4b5cf38d46 Sep 18 21:06:16.208450 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.208426 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbf713d_d757_4d4e_b94e_31ce8acaccd8.slice/crio-2ef45132d36fccc2a8badb79a95ce34be017df69828d679ff61fdd33f085c7ef WatchSource:0}: Error finding container 2ef45132d36fccc2a8badb79a95ce34be017df69828d679ff61fdd33f085c7ef: Status 404 returned error can't find the container with id 2ef45132d36fccc2a8badb79a95ce34be017df69828d679ff61fdd33f085c7ef Sep 18 21:06:16.209227 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.209210 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bef309_7448_4e16_a947_c916df0fb672.slice/crio-8c1760b12907c7ed9148cef6a96dcca3669f112ec800f7af1b3f944e9e38187b WatchSource:0}: Error finding container 8c1760b12907c7ed9148cef6a96dcca3669f112ec800f7af1b3f944e9e38187b: Status 404 returned error can't find the container with id 8c1760b12907c7ed9148cef6a96dcca3669f112ec800f7af1b3f944e9e38187b Sep 18 21:06:16.210563 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.210192 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79faf647_1593_440d_b79d_383738434d06.slice/crio-d5102f67706709032890852ec0ea3563b1496e5fefcd8d5644de47760ec01802 WatchSource:0}: Error finding container d5102f67706709032890852ec0ea3563b1496e5fefcd8d5644de47760ec01802: Status 404 returned error can't find the container with id d5102f67706709032890852ec0ea3563b1496e5fefcd8d5644de47760ec01802 Sep 18 21:06:16.212291 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:16.211543 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c22257_3ecd_4264_89ef_560ecd2a6c41.slice/crio-bc06f86584e4d118f193e13f02ee6c3322f03678c27776d397083ae7f90bcde6 WatchSource:0}: Error finding container bc06f86584e4d118f193e13f02ee6c3322f03678c27776d397083ae7f90bcde6: Status 404 returned error can't find the container with id bc06f86584e4d118f193e13f02ee6c3322f03678c27776d397083ae7f90bcde6 Sep 18 21:06:16.258767 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.258729 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:01:15 +0000 UTC" deadline="2028-03-05 13:15:52.849301157 +0000 UTC" Sep 18 21:06:16.258767 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.258764 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12808h9m36.590541163s" Sep 18 21:06:16.361766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.361570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"8c1760b12907c7ed9148cef6a96dcca3669f112ec800f7af1b3f944e9e38187b"} Sep 18 21:06:16.363630 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.363599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" event={"ID":"7dbf713d-d757-4d4e-b94e-31ce8acaccd8","Type":"ContainerStarted","Data":"2ef45132d36fccc2a8badb79a95ce34be017df69828d679ff61fdd33f085c7ef"} Sep 18 21:06:16.365820 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.365792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hptln" event={"ID":"bf72364b-d85e-432b-b5b9-6ea60f72f68f","Type":"ContainerStarted","Data":"4026c2012ed9c3d1fdf59a1083dffcc565cba2a86014afaafffbeb4b5cf38d46"} Sep 18 21:06:16.367709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.367686 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cr7wf" event={"ID":"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc","Type":"ContainerStarted","Data":"e0337653dc224ccb21c355b74a94ef5e5e78abba7c717329142f94d25ccf0410"} Sep 18 21:06:16.369726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.369701 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" event={"ID":"8a2162d0f83354e882cf0ecc5729b614","Type":"ContainerStarted","Data":"1d5f417c64e41bb14d0bfd84002248246881a3e8b06c1c6df5bae5b0188e9f52"} Sep 18 21:06:16.371019 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.370983 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9zlx2" event={"ID":"23c22257-3ecd-4264-89ef-560ecd2a6c41","Type":"ContainerStarted","Data":"bc06f86584e4d118f193e13f02ee6c3322f03678c27776d397083ae7f90bcde6"} Sep 18 21:06:16.372185 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.372161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4588v" event={"ID":"79faf647-1593-440d-b79d-383738434d06","Type":"ContainerStarted","Data":"d5102f67706709032890852ec0ea3563b1496e5fefcd8d5644de47760ec01802"} Sep 18 21:06:16.373215 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.373192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-55w59" event={"ID":"69f29fa6-0d0c-487a-ba87-8b99fbf8d364","Type":"ContainerStarted","Data":"a3d1ea65f498a755158e4a375181de5a28ac7bbf94b92b1e1ff523a771e5cf3b"} Sep 18 21:06:16.374691 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.374668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4j4" event={"ID":"34f89201-910a-440a-a3a1-e33ded838c8a","Type":"ContainerStarted","Data":"b757cb1037e532f8e4e3b5681d728633ae471cb6846f3296ffa30a00038a5504"} Sep 18 21:06:16.378503 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.376317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gxlff" event={"ID":"c61c453c-8a93-4f00-8856-74528a555127","Type":"ContainerStarted","Data":"99581a015b3031939ae6200363dc62cbe6f8bd9c05a00872bf9f10b260c60caf"} Sep 18 21:06:16.388972 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.388887 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-187.ec2.internal" podStartSLOduration=2.388865763 podStartE2EDuration="2.388865763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:16.388039563 +0000 UTC m=+5.317785449" watchObservedRunningTime="2026-09-18 21:06:16.388865763 +0000 UTC m=+5.318611695" Sep 18 21:06:16.836667 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.836628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:16.836856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.836702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:16.836931 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:16.836914 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:16.837016 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:16.837001 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret podName:66d8cb46-3ba4-4b18-b769-a7667c20495c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:18.83697935 +0000 UTC m=+7.766725218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret") pod "global-pull-secret-syncer-6xgxk" (UID: "66d8cb46-3ba4-4b18-b769-a7667c20495c") : object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:16.837521 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:16.837485 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:16.837640 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:16.837548 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs podName:9e35eaad-1b7d-40d2-a280-ef6d3c538a8c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:18.837533315 +0000 UTC m=+7.767279181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs") pod "network-metrics-daemon-rt75r" (UID: "9e35eaad-1b7d-40d2-a280-ef6d3c538a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:16.937114 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:16.937074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:16.937312 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:16.937297 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 21:06:16.937377 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:16.937320 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 21:06:16.937377 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:16.937333 2577 projected.go:196] Error preparing data for projected volume kube-api-access-h7nnk for pod openshift-network-diagnostics/network-check-target-mb586: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:16.937475 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:16.937400 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk podName:eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6 nodeName:}" failed. No retries permitted until 2026-09-18 21:06:18.937374909 +0000 UTC m=+7.867120773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7nnk" (UniqueName: "kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk") pod "network-check-target-mb586" (UID: "eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:17.260000 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:17.259861 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:01:15 +0000 UTC" deadline="2028-04-01 14:48:47.65232705 +0000 UTC" Sep 18 21:06:17.260000 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:17.259930 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13457h42m30.392401369s" Sep 18 21:06:17.353885 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:17.353086 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:17.353885 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:17.353214 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:17.353885 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:17.353591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:17.353885 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:17.353692 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:17.353885 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:17.353756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:17.353885 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:17.353845 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:17.390359 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:17.390316 2577 generic.go:356] "Generic (PLEG): container finished" podID="1510db1b6cb9287988e62a0f9d001698" containerID="9c97065418581fec1243416ba3342d5f53d1d627d66dcdbaa33141cfe036d94f" exitCode=0 Sep 18 21:06:17.390889 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:17.390863 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" event={"ID":"1510db1b6cb9287988e62a0f9d001698","Type":"ContainerDied","Data":"9c97065418581fec1243416ba3342d5f53d1d627d66dcdbaa33141cfe036d94f"} Sep 18 21:06:18.409658 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:18.409605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" event={"ID":"1510db1b6cb9287988e62a0f9d001698","Type":"ContainerStarted","Data":"d19a4e16d7302aa2956ba179a9900b9700c8189251ce6afbcd0a4730d8b9e18b"} Sep 18 21:06:18.853988 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:18.853933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:18.854181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:18.854041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:18.854181 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:18.854166 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:18.854303 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:18.854228 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs podName:9e35eaad-1b7d-40d2-a280-ef6d3c538a8c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:22.8542106 +0000 UTC m=+11.783956459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs") pod "network-metrics-daemon-rt75r" (UID: "9e35eaad-1b7d-40d2-a280-ef6d3c538a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:18.854627 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:18.854609 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:18.854698 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:18.854659 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret podName:66d8cb46-3ba4-4b18-b769-a7667c20495c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:22.854644658 +0000 UTC m=+11.784390520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret") pod "global-pull-secret-syncer-6xgxk" (UID: "66d8cb46-3ba4-4b18-b769-a7667c20495c") : object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:18.955293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:18.955253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:18.955486 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:18.955434 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 21:06:18.955486 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:18.955458 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 21:06:18.955486 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:18.955471 2577 projected.go:196] Error preparing data for projected volume kube-api-access-h7nnk for pod openshift-network-diagnostics/network-check-target-mb586: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:18.955650 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:18.955529 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk podName:eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6 nodeName:}" failed. No retries permitted until 2026-09-18 21:06:22.955511274 +0000 UTC m=+11.885257140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7nnk" (UniqueName: "kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk") pod "network-check-target-mb586" (UID: "eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:19.353106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:19.352531 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:19.353106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:19.352651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:19.353106 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:19.352671 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:19.353106 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:19.352732 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:19.353106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:19.352766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:19.353106 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:19.352839 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:21.353113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:21.353074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:21.353621 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:21.353208 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:21.353698 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:21.353640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:21.353772 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:21.353751 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:21.353836 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:21.353824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:21.353916 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:21.353898 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:22.890500 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:22.890455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:22.890996 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:22.890546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:22.890996 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:22.890662 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:22.890996 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:22.890723 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs podName:9e35eaad-1b7d-40d2-a280-ef6d3c538a8c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:30.890704192 +0000 UTC m=+19.820450050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs") pod "network-metrics-daemon-rt75r" (UID: "9e35eaad-1b7d-40d2-a280-ef6d3c538a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:22.891204 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:22.891134 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:22.891204 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:22.891188 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret podName:66d8cb46-3ba4-4b18-b769-a7667c20495c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:30.89117284 +0000 UTC m=+19.820918697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret") pod "global-pull-secret-syncer-6xgxk" (UID: "66d8cb46-3ba4-4b18-b769-a7667c20495c") : object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:22.991456 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:22.991409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:22.991650 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:22.991612 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 21:06:22.991650 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:22.991633 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 21:06:22.991650 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:22.991646 2577 projected.go:196] Error preparing data for projected volume kube-api-access-h7nnk for pod openshift-network-diagnostics/network-check-target-mb586: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:22.991805 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:22.991707 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk podName:eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6 nodeName:}" failed. No retries permitted until 2026-09-18 21:06:30.991687311 +0000 UTC m=+19.921433180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7nnk" (UniqueName: "kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk") pod "network-check-target-mb586" (UID: "eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:23.353868 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:23.353396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:23.353868 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:23.353501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:23.353868 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:23.353510 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:23.353868 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:23.353643 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:23.353868 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:23.353698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:23.353868 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:23.353769 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:25.352186 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:25.352150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:25.352741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:25.352208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:25.352741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:25.352150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:25.352741 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:25.352281 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:25.352741 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:25.352409 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:25.352741 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:25.352516 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:27.352936 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:27.352901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:27.353406 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:27.352950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:27.353406 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:27.352899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:27.353406 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:27.353069 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:27.353406 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:27.353122 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:27.353406 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:27.353197 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:29.356343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:29.356297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:29.356343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:29.356314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:29.356343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:29.356314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:29.356879 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:29.356413 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:29.356879 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:29.356552 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:29.356879 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:29.356654 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:30.958247 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:30.958198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:30.958686 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:30.958379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:30.958686 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:30.958412 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:30.958686 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:30.958484 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret podName:66d8cb46-3ba4-4b18-b769-a7667c20495c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:46.958463932 +0000 UTC m=+35.888209787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret") pod "global-pull-secret-syncer-6xgxk" (UID: "66d8cb46-3ba4-4b18-b769-a7667c20495c") : object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:30.958686 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:30.958501 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:30.958686 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:30.958553 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs podName:9e35eaad-1b7d-40d2-a280-ef6d3c538a8c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:46.958536368 +0000 UTC m=+35.888282227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs") pod "network-metrics-daemon-rt75r" (UID: "9e35eaad-1b7d-40d2-a280-ef6d3c538a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:31.059009 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:31.058967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:31.059220 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:31.059101 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 21:06:31.059220 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:31.059134 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 21:06:31.059220 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:31.059155 2577 projected.go:196] Error preparing data for projected volume kube-api-access-h7nnk for pod openshift-network-diagnostics/network-check-target-mb586: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:31.059220 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:31.059219 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk podName:eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6 nodeName:}" failed. No retries permitted until 2026-09-18 21:06:47.059201679 +0000 UTC m=+35.988947544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7nnk" (UniqueName: "kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk") pod "network-check-target-mb586" (UID: "eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:31.355697 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:31.355663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:31.355905 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:31.355665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:31.355905 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:31.355784 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:31.355905 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:31.355678 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:31.355905 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:31.355856 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:31.356108 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:31.355945 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:33.353565 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:33.353527 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:33.353939 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:33.353624 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:33.353939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:33.353700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:33.353939 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:33.353791 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:33.353939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:33.353828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:33.353939 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:33.353879 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:34.450407 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.450192 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:06:34.453618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.451184 2577 generic.go:356] "Generic (PLEG): container finished" podID="86bef309-7448-4e16-a947-c916df0fb672" containerID="05f49386d4a68816f00c39917cf982247a77dff49f63a3976789fed813feb92f" exitCode=1 Sep 18 21:06:34.453618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.451256 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"39b57238c3e324b043922d573125f3b5bba29850b3c450bb282e67a8f024ed57"} Sep 18 21:06:34.453618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.451421 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"204c4ad74a924aa5da1e5d7a0ea8281d6644ed3ea27f50a7fe0cfd3127a38406"} Sep 18 21:06:34.453618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.451489 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"0112d23237c61c2091e537cd72a748306a6c22d85ad48b5e2280791a542a37e7"} Sep 18 21:06:34.453618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.451558 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerDied","Data":"05f49386d4a68816f00c39917cf982247a77dff49f63a3976789fed813feb92f"} Sep 18 21:06:34.453618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.451625 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"223e0f04e091cc2d9ae7e0c672d351d0231180c47542e4c7d07ad401e71af720"} Sep 18 21:06:34.455017 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.454986 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" event={"ID":"7dbf713d-d757-4d4e-b94e-31ce8acaccd8","Type":"ContainerStarted","Data":"7d3491ec5542197f24642b925503fd17aaa4f312a2ca82cb9e7a4270ea68470a"} Sep 18 21:06:34.456548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.456525 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hptln" event={"ID":"bf72364b-d85e-432b-b5b9-6ea60f72f68f","Type":"ContainerStarted","Data":"57f9fae5fc8012584a8310685994d492ff89acc0e3e705690818d49307e02cd4"} Sep 18 21:06:34.457994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.457972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9zlx2" event={"ID":"23c22257-3ecd-4264-89ef-560ecd2a6c41","Type":"ContainerStarted","Data":"a2fa1c2c10292795e5932513f765be00123a890b2bbf25463ef5e1d8315c4c52"} Sep 18 21:06:34.459411 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.459379 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4588v" event={"ID":"79faf647-1593-440d-b79d-383738434d06","Type":"ContainerStarted","Data":"335d86237a50b42d270bc3c69cb77f30df2d074e8fc23df8550e3819efff048d"} Sep 18 21:06:34.460893 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.460871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-55w59" event={"ID":"69f29fa6-0d0c-487a-ba87-8b99fbf8d364","Type":"ContainerStarted","Data":"9fe4960488752c32811efb3df0a8cb488c4e23e6ab026acc505c60117e3ac880"} Sep 18 21:06:34.462693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.462670 2577 generic.go:356] "Generic (PLEG): container finished" podID="34f89201-910a-440a-a3a1-e33ded838c8a" containerID="7a77ab9b46422bd8f0a834e8ea7cbe9af8e909154e43372768919436fe7ffa10" exitCode=0 Sep 18 21:06:34.462786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.462756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4j4" event={"ID":"34f89201-910a-440a-a3a1-e33ded838c8a","Type":"ContainerDied","Data":"7a77ab9b46422bd8f0a834e8ea7cbe9af8e909154e43372768919436fe7ffa10"} Sep 18 21:06:34.464517 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.464494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gxlff" event={"ID":"c61c453c-8a93-4f00-8856-74528a555127","Type":"ContainerStarted","Data":"c4d69bd9ef79e0476bd74cff8cedbd2912e2ef181b6bf3e219384c90c06137d8"} Sep 18 21:06:34.479676 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.479600 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-187.ec2.internal" podStartSLOduration=20.4795864 podStartE2EDuration="20.4795864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:18.427042589 +0000 UTC m=+7.356788466" watchObservedRunningTime="2026-09-18 21:06:34.4795864 +0000 UTC m=+23.409332276" Sep 18 21:06:34.479913 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.479880 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hptln" podStartSLOduration=3.85575039 podStartE2EDuration="21.479870914s" totalImagesPullingTime="17.624120524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:34.479507248 +0000 UTC m=+23.409253124" watchObservedRunningTime="2026-09-18 21:06:34.479870914 +0000 UTC m=+23.409616788" Sep 18 21:06:34.526384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.526149 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9zlx2" podStartSLOduration=8.302878023 podStartE2EDuration="21.526128205s" totalImagesPullingTime="13.223250182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:34.499902125 +0000 UTC m=+23.429647998" watchObservedRunningTime="2026-09-18 21:06:34.526128205 +0000 UTC m=+23.455874165" Sep 18 21:06:34.547456 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.547405 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-55w59" podStartSLOduration=8.317520777 podStartE2EDuration="21.54739045s" totalImagesPullingTime="13.229869673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:34.547159626 +0000 UTC m=+23.476905499" watchObservedRunningTime="2026-09-18 21:06:34.54739045 +0000 UTC m=+23.477136322" Sep 18 21:06:34.576598 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.576545 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4588v" podStartSLOduration=4.018460719 podStartE2EDuration="21.576515115s" totalImagesPullingTime="17.558054396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:34.576051998 +0000 UTC m=+23.505797894" watchObservedRunningTime="2026-09-18 21:06:34.576515115 +0000 UTC m=+23.506260990" Sep 18 21:06:34.599873 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:34.599829 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gxlff" podStartSLOduration=4.030443552 podStartE2EDuration="21.599814457s" totalImagesPullingTime="17.569370905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:34.599664685 +0000 UTC m=+23.529410559" watchObservedRunningTime="2026-09-18 21:06:34.599814457 +0000 UTC m=+23.529560331" Sep 18 21:06:35.269019 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.268983 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 21:06:35.273061 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.272969 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T21:06:35.269003088Z","UUID":"71831ffe-8745-4396-a4a1-9f5211332d8e","Handler":null,"Name":"","Endpoint":""} Sep 18 21:06:35.274323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.274304 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 21:06:35.274438 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.274336 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 21:06:35.352930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.352898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:35.353130 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.352898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:35.353130 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.352898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:35.353130 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:35.353055 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:35.353130 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:35.353120 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:35.353319 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:35.353215 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:35.384920 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.384887 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:35.385461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.385430 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:35.470451 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.470418 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:06:35.470905 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.470867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"7471d04bc35f292a8678fc19e1adebddb338331e0ff6fc192d6d448018959952"} Sep 18 21:06:35.473134 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.473105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" event={"ID":"7dbf713d-d757-4d4e-b94e-31ce8acaccd8","Type":"ContainerStarted","Data":"bfc21c3a26afbf6713246896aa3f9d3b973deec1f6146054132d5c882e762b41"} Sep 18 21:06:35.474818 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.474785 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cr7wf" event={"ID":"f6dbdfbd-1dfb-4187-9a9b-fb67b32baabc","Type":"ContainerStarted","Data":"0dad25d71184b32d020f0e8575538f10d651a8a3ad62d1450d0f37593d566354"} Sep 18 21:06:35.475559 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.475539 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:35.476081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.476060 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-55w59" Sep 18 21:06:35.492855 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:35.492794 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cr7wf" podStartSLOduration=4.928580867 podStartE2EDuration="22.492775162s" totalImagesPullingTime="17.564194295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:35.49157578 +0000 UTC m=+24.421321635" watchObservedRunningTime="2026-09-18 21:06:35.492775162 +0000 UTC m=+24.422521035" Sep 18 21:06:36.479204 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:36.479121 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" event={"ID":"7dbf713d-d757-4d4e-b94e-31ce8acaccd8","Type":"ContainerStarted","Data":"7c7270c4ccc27edc7e89e13f4fe5c1b7baa879bc5b8939c5c34877160ffa68cc"} Sep 18 21:06:36.502741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:36.502693 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wtg4" podStartSLOduration=3.964823616 podStartE2EDuration="23.502676931s" totalImagesPullingTime="19.537853315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:36.502479506 +0000 UTC m=+25.432225379" watchObservedRunningTime="2026-09-18 21:06:36.502676931 +0000 UTC m=+25.432422849" Sep 18 21:06:37.352430 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:37.352396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:37.352598 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:37.352397 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:37.352598 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:37.352506 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:37.352598 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:37.352400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:37.352598 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:37.352589 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:37.352766 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:37.352679 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:37.484753 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:37.484726 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:06:37.485211 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:37.485184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"a9c376fb6abc6de4a12996b3a1368d09c046e5d2f1723ff1d31a8f3a6c73f819"} Sep 18 21:06:39.353059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.352862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:39.353732 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.352862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:39.353732 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:39.353135 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:39.353732 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:39.353235 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:39.353732 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.352862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:39.353732 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:39.353352 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:39.492491 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.492460 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:06:39.492883 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.492852 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"f0a9f7cd80f9f00a103aa176926d154bf62764204bb4cb828e9f82448470ccfc"} Sep 18 21:06:39.493153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.493133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:39.493312 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.493285 2577 scope.go:117] "RemoveContainer" containerID="05f49386d4a68816f00c39917cf982247a77dff49f63a3976789fed813feb92f" Sep 18 21:06:39.494901 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.494877 2577 generic.go:356] "Generic (PLEG): container finished" podID="34f89201-910a-440a-a3a1-e33ded838c8a" containerID="e5bcce9259d815a2785017d6be8c209108fc75fdf2259720d6ed7f3b93d008e9" exitCode=0 Sep 18 21:06:39.495018 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.494908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4j4" event={"ID":"34f89201-910a-440a-a3a1-e33ded838c8a","Type":"ContainerDied","Data":"e5bcce9259d815a2785017d6be8c209108fc75fdf2259720d6ed7f3b93d008e9"} Sep 18 21:06:39.509371 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:39.509226 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:40.500606 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.500433 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:06:40.501083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.501014 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" event={"ID":"86bef309-7448-4e16-a947-c916df0fb672","Type":"ContainerStarted","Data":"a665479b30da2b4795bd6b85d14776260e085195e2bc63606442ae7ef55ede6d"} Sep 18 21:06:40.501212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.501181 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:40.501344 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.501244 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:40.517218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.517193 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:06:40.534473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.534423 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" podStartSLOduration=9.957103665 podStartE2EDuration="27.534405763s" totalImagesPullingTime="17.577302098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:40.533774878 +0000 UTC m=+29.463520751" watchObservedRunningTime="2026-09-18 21:06:40.534405763 +0000 UTC m=+29.464151635" Sep 18 21:06:40.926798 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.926766 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6xgxk"] Sep 18 21:06:40.926996 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.926888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:40.926996 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:40.926978 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:40.933500 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.933471 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mb586"] Sep 18 21:06:40.933648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.933609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:40.933743 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:40.933723 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:40.934028 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.934010 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rt75r"] Sep 18 21:06:40.934114 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:40.934103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:40.934223 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:40.934196 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:41.505990 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:41.505946 2577 generic.go:356] "Generic (PLEG): container finished" podID="34f89201-910a-440a-a3a1-e33ded838c8a" containerID="b6f76530dce104d8985a590191e73a47f6126012ae2bf62b7cebdaf853209fe5" exitCode=0 Sep 18 21:06:41.506380 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:41.506020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4j4" event={"ID":"34f89201-910a-440a-a3a1-e33ded838c8a","Type":"ContainerDied","Data":"b6f76530dce104d8985a590191e73a47f6126012ae2bf62b7cebdaf853209fe5"} Sep 18 21:06:42.352248 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:42.352211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:42.352467 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:42.352319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:42.352467 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:42.352332 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:42.352467 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:42.352346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:42.352467 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:42.352418 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:42.352659 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:42.352497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:43.512743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:43.512709 2577 generic.go:356] "Generic (PLEG): container finished" podID="34f89201-910a-440a-a3a1-e33ded838c8a" containerID="42c29b0773b12f716a784605dd3899be315987ae30b5592c568ef61f683a4d97" exitCode=0 Sep 18 21:06:43.513206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:43.512773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4j4" event={"ID":"34f89201-910a-440a-a3a1-e33ded838c8a","Type":"ContainerDied","Data":"42c29b0773b12f716a784605dd3899be315987ae30b5592c568ef61f683a4d97"} Sep 18 21:06:44.352459 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:44.352280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:44.352636 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:44.352281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:44.352636 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:44.352338 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:44.352730 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:44.352627 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:44.352730 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:44.352633 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:44.352730 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:44.352696 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:46.352607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.352567 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:46.353093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.352625 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:46.353093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.352690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:46.353093 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:46.352707 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mb586" podUID="eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6" Sep 18 21:06:46.353093 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:46.352784 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xgxk" podUID="66d8cb46-3ba4-4b18-b769-a7667c20495c" Sep 18 21:06:46.353093 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:46.352902 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt75r" podUID="9e35eaad-1b7d-40d2-a280-ef6d3c538a8c" Sep 18 21:06:46.898555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.898520 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-187.ec2.internal" event="NodeReady" Sep 18 21:06:46.898773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.898693 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 18 21:06:46.958134 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.958101 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hpqpv"] Sep 18 21:06:46.971181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.971146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:46.971371 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.971207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:46.971371 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:46.971312 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:46.971490 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:46.971382 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs podName:9e35eaad-1b7d-40d2-a280-ef6d3c538a8c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:18.971367302 +0000 UTC m=+67.901113166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs") pod "network-metrics-daemon-rt75r" (UID: "9e35eaad-1b7d-40d2-a280-ef6d3c538a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 21:06:46.971490 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:46.971322 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:46.971490 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:46.971459 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret podName:66d8cb46-3ba4-4b18-b769-a7667c20495c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:18.971441603 +0000 UTC m=+67.901187472 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret") pod "global-pull-secret-syncer-6xgxk" (UID: "66d8cb46-3ba4-4b18-b769-a7667c20495c") : object "kube-system"/"original-pull-secret" not registered Sep 18 21:06:46.983026 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.983001 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dvqc2"] Sep 18 21:06:46.983189 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.983160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:46.986144 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.986121 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4xcf\"" Sep 18 21:06:46.986566 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.986531 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 21:06:46.986687 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.986664 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 21:06:46.997015 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.996992 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hpqpv"] Sep 18 21:06:46.997126 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.997038 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dvqc2"] Sep 18 21:06:46.997179 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:46.997130 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dvqc2" Sep 18 21:06:47.001222 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.001203 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 21:06:47.001424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.001394 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 21:06:47.001528 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.001458 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 21:06:47.001605 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.001538 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jdg2h\"" Sep 18 21:06:47.072550 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.072500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vcj5\" (UniqueName: \"kubernetes.io/projected/f1aca64a-306c-4902-94c5-0f3845ac290b-kube-api-access-8vcj5\") pod \"ingress-canary-dvqc2\" (UID: \"f1aca64a-306c-4902-94c5-0f3845ac290b\") " pod="openshift-ingress-canary/ingress-canary-dvqc2" Sep 18 21:06:47.072550 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.072552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5s7d\" (UniqueName: \"kubernetes.io/projected/b085f059-d211-47ed-9747-bb25ea39e37d-kube-api-access-q5s7d\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.072796 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.072618 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b085f059-d211-47ed-9747-bb25ea39e37d-config-volume\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.072796 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.072680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:47.072796 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.072744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1aca64a-306c-4902-94c5-0f3845ac290b-cert\") pod \"ingress-canary-dvqc2\" (UID: \"f1aca64a-306c-4902-94c5-0f3845ac290b\") " pod="openshift-ingress-canary/ingress-canary-dvqc2" Sep 18 21:06:47.072796 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:47.072772 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 21:06:47.072796 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.072787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.073022 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.072814 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b085f059-d211-47ed-9747-bb25ea39e37d-tmp-dir\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.073022 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:47.072793 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 21:06:47.073022 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:47.072858 2577 projected.go:196] Error preparing data for projected volume kube-api-access-h7nnk for pod openshift-network-diagnostics/network-check-target-mb586: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:47.073022 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:47.072913 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk podName:eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6 nodeName:}" failed. No retries permitted until 2026-09-18 21:07:19.072895041 +0000 UTC m=+68.002640909 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7nnk" (UniqueName: "kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk") pod "network-check-target-mb586" (UID: "eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 21:06:47.173232 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.173190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b085f059-d211-47ed-9747-bb25ea39e37d-config-volume\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.173410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.173288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1aca64a-306c-4902-94c5-0f3845ac290b-cert\") pod \"ingress-canary-dvqc2\" (UID: \"f1aca64a-306c-4902-94c5-0f3845ac290b\") " pod="openshift-ingress-canary/ingress-canary-dvqc2" Sep 18 21:06:47.173410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.173331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.173410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.173356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b085f059-d211-47ed-9747-bb25ea39e37d-tmp-dir\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.173592 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:47.173473 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 21:06:47.173592 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:47.173556 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls podName:b085f059-d211-47ed-9747-bb25ea39e37d nodeName:}" failed. No retries permitted until 2026-09-18 21:06:47.673526395 +0000 UTC m=+36.603272271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls") pod "dns-default-hpqpv" (UID: "b085f059-d211-47ed-9747-bb25ea39e37d") : secret "dns-default-metrics-tls" not found Sep 18 21:06:47.173592 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.173581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vcj5\" (UniqueName: \"kubernetes.io/projected/f1aca64a-306c-4902-94c5-0f3845ac290b-kube-api-access-8vcj5\") pod \"ingress-canary-dvqc2\" (UID: \"f1aca64a-306c-4902-94c5-0f3845ac290b\") " pod="openshift-ingress-canary/ingress-canary-dvqc2" Sep 18 21:06:47.173740 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.173624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5s7d\" (UniqueName: \"kubernetes.io/projected/b085f059-d211-47ed-9747-bb25ea39e37d-kube-api-access-q5s7d\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.173740 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.173691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b085f059-d211-47ed-9747-bb25ea39e37d-tmp-dir\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.173875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.173846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b085f059-d211-47ed-9747-bb25ea39e37d-config-volume\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.177945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.177923 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1aca64a-306c-4902-94c5-0f3845ac290b-cert\") pod \"ingress-canary-dvqc2\" (UID: \"f1aca64a-306c-4902-94c5-0f3845ac290b\") " pod="openshift-ingress-canary/ingress-canary-dvqc2" Sep 18 21:06:47.182205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.182176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5s7d\" (UniqueName: \"kubernetes.io/projected/b085f059-d211-47ed-9747-bb25ea39e37d-kube-api-access-q5s7d\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.182318 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.182227 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vcj5\" (UniqueName: \"kubernetes.io/projected/f1aca64a-306c-4902-94c5-0f3845ac290b-kube-api-access-8vcj5\") pod \"ingress-canary-dvqc2\" (UID: \"f1aca64a-306c-4902-94c5-0f3845ac290b\") " pod="openshift-ingress-canary/ingress-canary-dvqc2" Sep 18 21:06:47.306924 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.306886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dvqc2" Sep 18 21:06:47.676936 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.676857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:47.677653 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:47.677034 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 21:06:47.677653 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:47.677099 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls podName:b085f059-d211-47ed-9747-bb25ea39e37d nodeName:}" failed. No retries permitted until 2026-09-18 21:06:48.677081643 +0000 UTC m=+37.606827511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls") pod "dns-default-hpqpv" (UID: "b085f059-d211-47ed-9747-bb25ea39e37d") : secret "dns-default-metrics-tls" not found Sep 18 21:06:47.762373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:47.762262 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dvqc2"] Sep 18 21:06:47.767978 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:06:47.767928 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1aca64a_306c_4902_94c5_0f3845ac290b.slice/crio-d426742679b66f271ae03a7c8dff122bb9124efc6f50dd5c4eb458e956b78932 WatchSource:0}: Error finding container d426742679b66f271ae03a7c8dff122bb9124efc6f50dd5c4eb458e956b78932: Status 404 returned error can't find the container with id d426742679b66f271ae03a7c8dff122bb9124efc6f50dd5c4eb458e956b78932 Sep 18 21:06:48.353107 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.353069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:06:48.353369 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.353124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:06:48.353369 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.353069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:06:48.355874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.355852 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 21:06:48.356852 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.356811 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xrfdl\"" Sep 18 21:06:48.356852 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.356820 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 21:06:48.356852 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.356832 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 21:06:48.356852 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.356847 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6vrc\"" Sep 18 21:06:48.357185 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.356824 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 21:06:48.528445 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.528403 2577 generic.go:356] "Generic (PLEG): container finished" podID="34f89201-910a-440a-a3a1-e33ded838c8a" containerID="64d62a627dc096741e311339e7120a4e58be223029a076620abf222e31e1d530" exitCode=0 Sep 18 21:06:48.528614 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.528437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4j4" event={"ID":"34f89201-910a-440a-a3a1-e33ded838c8a","Type":"ContainerDied","Data":"64d62a627dc096741e311339e7120a4e58be223029a076620abf222e31e1d530"} Sep 18 21:06:48.529893 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.529868 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dvqc2" event={"ID":"f1aca64a-306c-4902-94c5-0f3845ac290b","Type":"ContainerStarted","Data":"d426742679b66f271ae03a7c8dff122bb9124efc6f50dd5c4eb458e956b78932"} Sep 18 21:06:48.684659 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:48.684504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:48.684659 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:48.684608 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 21:06:48.685097 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:48.684873 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls podName:b085f059-d211-47ed-9747-bb25ea39e37d nodeName:}" failed. No retries permitted until 2026-09-18 21:06:50.684642644 +0000 UTC m=+39.614388514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls") pod "dns-default-hpqpv" (UID: "b085f059-d211-47ed-9747-bb25ea39e37d") : secret "dns-default-metrics-tls" not found Sep 18 21:06:49.536838 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:49.536792 2577 generic.go:356] "Generic (PLEG): container finished" podID="34f89201-910a-440a-a3a1-e33ded838c8a" containerID="7d89347cbfeb7db7f3010805a40bec486aff8768c75e1acf73c0bd14ee2306fc" exitCode=0 Sep 18 21:06:49.537132 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:49.536870 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4j4" event={"ID":"34f89201-910a-440a-a3a1-e33ded838c8a","Type":"ContainerDied","Data":"7d89347cbfeb7db7f3010805a40bec486aff8768c75e1acf73c0bd14ee2306fc"} Sep 18 21:06:50.542705 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:50.542674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js4j4" event={"ID":"34f89201-910a-440a-a3a1-e33ded838c8a","Type":"ContainerStarted","Data":"30c4364e24ba72e737c7a07e6f5a002348df2f47631dbf0aa090b9ce75069956"} Sep 18 21:06:50.544199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:50.544175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dvqc2" event={"ID":"f1aca64a-306c-4902-94c5-0f3845ac290b","Type":"ContainerStarted","Data":"c19aa1fa7da2cd423f0fdb6aae042cddef9af069f05f6915fc3c6da0801cfdfc"} Sep 18 21:06:50.567218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:50.567129 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-js4j4" podStartSLOduration=9.202710737 podStartE2EDuration="37.56711177s" totalImagesPullingTime="28.364401033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:50.565719572 +0000 UTC m=+39.495465445" watchObservedRunningTime="2026-09-18 21:06:50.56711177 +0000 UTC m=+39.496857643" Sep 18 21:06:50.581698 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:50.581658 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dvqc2" podStartSLOduration=2.415914035 podStartE2EDuration="4.581645554s" totalImagesPullingTime="2.165731519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:50.581013354 +0000 UTC m=+39.510759227" watchObservedRunningTime="2026-09-18 21:06:50.581645554 +0000 UTC m=+39.511391426" Sep 18 21:06:50.701073 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:50.701022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:50.701275 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:50.701157 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 21:06:50.701275 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:50.701233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls podName:b085f059-d211-47ed-9747-bb25ea39e37d nodeName:}" failed. No retries permitted until 2026-09-18 21:06:54.701210335 +0000 UTC m=+43.630956211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls") pod "dns-default-hpqpv" (UID: "b085f059-d211-47ed-9747-bb25ea39e37d") : secret "dns-default-metrics-tls" not found Sep 18 21:06:54.729581 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:06:54.729536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:06:54.729989 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:54.729691 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 21:06:54.729989 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:06:54.729753 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls podName:b085f059-d211-47ed-9747-bb25ea39e37d nodeName:}" failed. No retries permitted until 2026-09-18 21:07:02.729737172 +0000 UTC m=+51.659483040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls") pod "dns-default-hpqpv" (UID: "b085f059-d211-47ed-9747-bb25ea39e37d") : secret "dns-default-metrics-tls" not found Sep 18 21:07:02.783060 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:02.783010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:07:02.783440 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:02.783167 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 21:07:02.783440 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:02.783244 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls podName:b085f059-d211-47ed-9747-bb25ea39e37d nodeName:}" failed. No retries permitted until 2026-09-18 21:07:18.783227144 +0000 UTC m=+67.712972994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls") pod "dns-default-hpqpv" (UID: "b085f059-d211-47ed-9747-bb25ea39e37d") : secret "dns-default-metrics-tls" not found Sep 18 21:07:12.528200 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:12.528167 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzszz" Sep 18 21:07:16.217603 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.217573 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-m8g7n"] Sep 18 21:07:16.221093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.221076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.223422 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.223396 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 18 21:07:16.223546 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.223507 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 18 21:07:16.223546 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.223518 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vnt4l\"" Sep 18 21:07:16.224297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.224280 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 21:07:16.224389 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.224325 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 21:07:16.233201 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.230262 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 18 21:07:16.233201 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.231950 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-m8g7n"] Sep 18 21:07:16.381862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.381802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2ed5b13a-605a-46fe-a982-2d16520f93ea-snapshots\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.381862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.381881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ed5b13a-605a-46fe-a982-2d16520f93ea-serving-cert\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.382105 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.381988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ed5b13a-605a-46fe-a982-2d16520f93ea-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.382105 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.382029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ed5b13a-605a-46fe-a982-2d16520f93ea-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.382105 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.382079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ed5b13a-605a-46fe-a982-2d16520f93ea-tmp\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.382105 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.382097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprjq\" (UniqueName: \"kubernetes.io/projected/2ed5b13a-605a-46fe-a982-2d16520f93ea-kube-api-access-wprjq\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.440428 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.440392 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67ff68f94b-j7d5n"] Sep 18 21:07:16.442298 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.442277 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k"] Sep 18 21:07:16.442426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.442405 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.444062 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.444047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.446771 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.446754 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 18 21:07:16.447820 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.447805 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 18 21:07:16.447913 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.447864 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 18 21:07:16.447913 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.447871 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d7nd7\"" Sep 18 21:07:16.448439 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.448424 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2mx85\"" Sep 18 21:07:16.448539 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.448478 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 18 21:07:16.450693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.450669 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 21:07:16.453079 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.453063 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 18 21:07:16.455285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.455265 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 18 21:07:16.455386 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.455319 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 21:07:16.464138 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.464110 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k"] Sep 18 21:07:16.467548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.467523 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67ff68f94b-j7d5n"] Sep 18 21:07:16.483275 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.483198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ed5b13a-605a-46fe-a982-2d16520f93ea-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.483275 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.483250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ed5b13a-605a-46fe-a982-2d16520f93ea-tmp\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.483443 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.483276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprjq\" (UniqueName: \"kubernetes.io/projected/2ed5b13a-605a-46fe-a982-2d16520f93ea-kube-api-access-wprjq\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.483443 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.483333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2ed5b13a-605a-46fe-a982-2d16520f93ea-snapshots\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.483443 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.483382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ed5b13a-605a-46fe-a982-2d16520f93ea-serving-cert\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.483443 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.483417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ed5b13a-605a-46fe-a982-2d16520f93ea-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.483698 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.483680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ed5b13a-605a-46fe-a982-2d16520f93ea-tmp\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.483996 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.483981 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2ed5b13a-605a-46fe-a982-2d16520f93ea-snapshots\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.484043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.484019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ed5b13a-605a-46fe-a982-2d16520f93ea-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.484229 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.484212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ed5b13a-605a-46fe-a982-2d16520f93ea-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.486646 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.486620 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ed5b13a-605a-46fe-a982-2d16520f93ea-serving-cert\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.497582 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.497546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprjq\" (UniqueName: \"kubernetes.io/projected/2ed5b13a-605a-46fe-a982-2d16520f93ea-kube-api-access-wprjq\") pod \"insights-operator-6f9c644dd9-m8g7n\" (UID: \"2ed5b13a-605a-46fe-a982-2d16520f93ea\") " pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.530073 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.530038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" Sep 18 21:07:16.584426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-image-registry-private-configuration\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.584426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-certificates\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.584577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-bound-sa-token\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.584577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.584577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.584577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmwgv\" (UniqueName: \"kubernetes.io/projected/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-kube-api-access-bmwgv\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.584766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-trusted-ca\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.584766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-installation-pull-secrets\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.584766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7bg\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-kube-api-access-gr7bg\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.584911 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.584911 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.584840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6faf6bb-009c-44d1-b93f-197543ed278c-ca-trust-extracted\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686048 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-image-registry-private-configuration\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686048 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-certificates\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686302 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-bound-sa-token\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686302 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.686405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.686405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmwgv\" (UniqueName: \"kubernetes.io/projected/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-kube-api-access-bmwgv\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.686405 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:16.686363 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:16.686405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-trusted-ca\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-installation-pull-secrets\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686720 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:16.686434 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls podName:b4311bed-80a0-4146-9ec8-ecd4daedaf0c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:17.18641503 +0000 UTC m=+66.116160894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-kls8k" (UID: "b4311bed-80a0-4146-9ec8-ecd4daedaf0c") : secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:16.686720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7bg\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-kube-api-access-gr7bg\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6faf6bb-009c-44d1-b93f-197543ed278c-ca-trust-extracted\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.686720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-certificates\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.687016 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:16.686773 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 21:07:16.687016 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:16.686791 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ff68f94b-j7d5n: secret "image-registry-tls" not found Sep 18 21:07:16.687016 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:16.686835 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls podName:d6faf6bb-009c-44d1-b93f-197543ed278c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:17.18682099 +0000 UTC m=+66.116566863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls") pod "image-registry-67ff68f94b-j7d5n" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c") : secret "image-registry-tls" not found Sep 18 21:07:16.687016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.686917 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6faf6bb-009c-44d1-b93f-197543ed278c-ca-trust-extracted\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.687216 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.687082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.687350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.687321 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-trusted-ca\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.688588 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.688572 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-image-registry-private-configuration\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.688937 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.688907 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-installation-pull-secrets\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.697419 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.697397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7bg\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-kube-api-access-gr7bg\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.697777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.697753 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmwgv\" (UniqueName: \"kubernetes.io/projected/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-kube-api-access-bmwgv\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:16.698676 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.698655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-bound-sa-token\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:16.801580 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:16.801496 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-m8g7n"] Sep 18 21:07:16.804041 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:16.804013 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed5b13a_605a_46fe_a982_2d16520f93ea.slice/crio-0755b4280b0c102312fcf24ad558afee910a67ae9834e4b71066c40b3f6a49a0 WatchSource:0}: Error finding container 0755b4280b0c102312fcf24ad558afee910a67ae9834e4b71066c40b3f6a49a0: Status 404 returned error can't find the container with id 0755b4280b0c102312fcf24ad558afee910a67ae9834e4b71066c40b3f6a49a0 Sep 18 21:07:17.191340 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:17.191304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:17.191502 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:17.191366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:17.191502 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:17.191469 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 21:07:17.191502 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:17.191487 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:17.191613 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:17.191548 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls podName:b4311bed-80a0-4146-9ec8-ecd4daedaf0c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:18.191534965 +0000 UTC m=+67.121280830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-kls8k" (UID: "b4311bed-80a0-4146-9ec8-ecd4daedaf0c") : secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:17.191613 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:17.191492 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ff68f94b-j7d5n: secret "image-registry-tls" not found Sep 18 21:07:17.191613 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:17.191605 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls podName:d6faf6bb-009c-44d1-b93f-197543ed278c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:18.191591116 +0000 UTC m=+67.121336967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls") pod "image-registry-67ff68f94b-j7d5n" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c") : secret "image-registry-tls" not found Sep 18 21:07:17.604531 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:17.604501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" event={"ID":"2ed5b13a-605a-46fe-a982-2d16520f93ea","Type":"ContainerStarted","Data":"0755b4280b0c102312fcf24ad558afee910a67ae9834e4b71066c40b3f6a49a0"} Sep 18 21:07:18.198998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:18.198942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:18.199182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:18.199030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:18.199182 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:18.199102 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 21:07:18.199182 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:18.199119 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:18.199182 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:18.199126 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ff68f94b-j7d5n: secret "image-registry-tls" not found Sep 18 21:07:18.199343 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:18.199188 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls podName:b4311bed-80a0-4146-9ec8-ecd4daedaf0c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:20.199166826 +0000 UTC m=+69.128912679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-kls8k" (UID: "b4311bed-80a0-4146-9ec8-ecd4daedaf0c") : secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:18.199343 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:18.199209 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls podName:d6faf6bb-009c-44d1-b93f-197543ed278c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:20.199198359 +0000 UTC m=+69.128944216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls") pod "image-registry-67ff68f94b-j7d5n" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c") : secret "image-registry-tls" not found Sep 18 21:07:18.803863 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:18.803824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:07:18.804274 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:18.803993 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 21:07:18.804274 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:18.804057 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls podName:b085f059-d211-47ed-9747-bb25ea39e37d nodeName:}" failed. No retries permitted until 2026-09-18 21:07:50.804038396 +0000 UTC m=+99.733784267 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls") pod "dns-default-hpqpv" (UID: "b085f059-d211-47ed-9747-bb25ea39e37d") : secret "dns-default-metrics-tls" not found Sep 18 21:07:19.005827 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.005792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:07:19.006034 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.005850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:07:19.008297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.008279 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 21:07:19.008350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.008321 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 21:07:19.016144 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:19.016120 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 21:07:19.016249 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:19.016202 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs podName:9e35eaad-1b7d-40d2-a280-ef6d3c538a8c nodeName:}" failed. No retries permitted until 2026-09-18 21:08:23.016184115 +0000 UTC m=+131.945929986 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs") pod "network-metrics-daemon-rt75r" (UID: "9e35eaad-1b7d-40d2-a280-ef6d3c538a8c") : secret "metrics-daemon-secret" not found Sep 18 21:07:19.018256 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.018236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66d8cb46-3ba4-4b18-b769-a7667c20495c-original-pull-secret\") pod \"global-pull-secret-syncer-6xgxk\" (UID: \"66d8cb46-3ba4-4b18-b769-a7667c20495c\") " pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:07:19.106661 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.106621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:07:19.109439 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.109414 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 21:07:19.119985 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.119944 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 21:07:19.130567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.130541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7nnk\" (UniqueName: \"kubernetes.io/projected/eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6-kube-api-access-h7nnk\") pod \"network-check-target-mb586\" (UID: \"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6\") " pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:07:19.275473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.275437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xrfdl\"" Sep 18 21:07:19.277236 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.277216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xgxk" Sep 18 21:07:19.284218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.284199 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:07:19.547866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.547788 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6xgxk"] Sep 18 21:07:19.551117 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:19.551076 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d8cb46_3ba4_4b18_b769_a7667c20495c.slice/crio-efa458a6c9cd2daa8e5da9b73c76e5b95b79b82cc912aba471dc0a724f869be3 WatchSource:0}: Error finding container efa458a6c9cd2daa8e5da9b73c76e5b95b79b82cc912aba471dc0a724f869be3: Status 404 returned error can't find the container with id efa458a6c9cd2daa8e5da9b73c76e5b95b79b82cc912aba471dc0a724f869be3 Sep 18 21:07:19.565546 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.565521 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mb586"] Sep 18 21:07:19.568082 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:19.568060 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0a4e2b_b05e_49fb_b463_7bf6bd793eb6.slice/crio-7e138e1612f8d16cd2c4c6388d60b6d31aa83bfd081af07c772f4db24985b8af WatchSource:0}: Error finding container 7e138e1612f8d16cd2c4c6388d60b6d31aa83bfd081af07c772f4db24985b8af: Status 404 returned error can't find the container with id 7e138e1612f8d16cd2c4c6388d60b6d31aa83bfd081af07c772f4db24985b8af Sep 18 21:07:19.611272 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.611234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6xgxk" event={"ID":"66d8cb46-3ba4-4b18-b769-a7667c20495c","Type":"ContainerStarted","Data":"efa458a6c9cd2daa8e5da9b73c76e5b95b79b82cc912aba471dc0a724f869be3"} Sep 18 21:07:19.612750 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.612725 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" event={"ID":"2ed5b13a-605a-46fe-a982-2d16520f93ea","Type":"ContainerStarted","Data":"4ffe9aa7c757268d22ba4d2cbba03f609b2aa637e49476b9f34cea81538d6103"} Sep 18 21:07:19.613968 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.613937 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mb586" event={"ID":"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6","Type":"ContainerStarted","Data":"7e138e1612f8d16cd2c4c6388d60b6d31aa83bfd081af07c772f4db24985b8af"} Sep 18 21:07:19.629087 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:19.629042 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" podStartSLOduration=1.5204704740000001 podStartE2EDuration="3.629024623s" totalImagesPullingTime="2.108554149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:19.62813986 +0000 UTC m=+68.557885724" watchObservedRunningTime="2026-09-18 21:07:19.629024623 +0000 UTC m=+68.558770536" Sep 18 21:07:20.215814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:20.215775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:20.216314 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:20.215864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:20.216314 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:20.215971 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 21:07:20.216314 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:20.215999 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ff68f94b-j7d5n: secret "image-registry-tls" not found Sep 18 21:07:20.216314 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:20.216016 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:20.216314 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:20.216073 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls podName:b4311bed-80a0-4146-9ec8-ecd4daedaf0c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:24.216053925 +0000 UTC m=+73.145799782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-kls8k" (UID: "b4311bed-80a0-4146-9ec8-ecd4daedaf0c") : secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:20.216314 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:20.216090 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls podName:d6faf6bb-009c-44d1-b93f-197543ed278c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:24.216081178 +0000 UTC m=+73.145827036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls") pod "image-registry-67ff68f94b-j7d5n" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c") : secret "image-registry-tls" not found Sep 18 21:07:21.593675 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.593635 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cqmw5"] Sep 18 21:07:21.596866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.596828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.599967 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.599710 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lfhrj\"" Sep 18 21:07:21.600103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.599973 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 21:07:21.600565 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.600530 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 21:07:21.620579 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.620534 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cqmw5"] Sep 18 21:07:21.629000 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.628967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-data-volume\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.629171 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.629116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.629234 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.629180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-crio-socket\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.629234 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.629217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.629342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.629241 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5vwx\" (UniqueName: \"kubernetes.io/projected/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-api-access-g5vwx\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.729649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.729606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-data-volume\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.729849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.729711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.729849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.729756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-crio-socket\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.729849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.729791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.729849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.729815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5vwx\" (UniqueName: \"kubernetes.io/projected/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-api-access-g5vwx\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.730103 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:21.729880 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 21:07:21.730103 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:21.730006 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 21:07:21.730103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.730043 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-data-volume\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.730103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.730064 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-crio-socket\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:21.730305 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:21.730011 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:22.229935309 +0000 UTC m=+71.159681182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : secret "insights-runtime-extractor-tls" not found Sep 18 21:07:21.730305 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:21.730151 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:22.230121067 +0000 UTC m=+71.159866925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : configmap "kube-rbac-proxy" not found Sep 18 21:07:21.739096 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:21.739031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5vwx\" (UniqueName: \"kubernetes.io/projected/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-api-access-g5vwx\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:22.233865 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:22.233825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:22.234052 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:22.233898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:22.234052 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:22.234002 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 21:07:22.234052 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:22.234038 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 21:07:22.234191 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:22.234067 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:23.234050047 +0000 UTC m=+72.163795902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : secret "insights-runtime-extractor-tls" not found Sep 18 21:07:22.234191 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:22.234105 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:23.234085251 +0000 UTC m=+72.163831127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : configmap "kube-rbac-proxy" not found Sep 18 21:07:23.242443 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:23.242399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:23.242912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:23.242482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:23.242912 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:23.242618 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 21:07:23.242912 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:23.242635 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 21:07:23.242912 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:23.242700 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:25.242680723 +0000 UTC m=+74.172426599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : secret "insights-runtime-extractor-tls" not found Sep 18 21:07:23.242912 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:23.242725 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:25.242714865 +0000 UTC m=+74.172460717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : configmap "kube-rbac-proxy" not found Sep 18 21:07:24.120424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:24.120396 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4588v_79faf647-1593-440d-b79d-383738434d06/dns-node-resolver/0.log" Sep 18 21:07:24.250971 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:24.250786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:24.250971 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:24.250892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:24.250971 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:24.250941 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:24.251389 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:24.251028 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls podName:b4311bed-80a0-4146-9ec8-ecd4daedaf0c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:32.251013235 +0000 UTC m=+81.180759086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-kls8k" (UID: "b4311bed-80a0-4146-9ec8-ecd4daedaf0c") : secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:24.251389 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:24.251053 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 21:07:24.251389 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:24.251070 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ff68f94b-j7d5n: secret "image-registry-tls" not found Sep 18 21:07:24.251389 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:24.251121 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls podName:d6faf6bb-009c-44d1-b93f-197543ed278c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:32.251106382 +0000 UTC m=+81.180852233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls") pod "image-registry-67ff68f94b-j7d5n" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c") : secret "image-registry-tls" not found Sep 18 21:07:24.632525 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:24.632490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6xgxk" event={"ID":"66d8cb46-3ba4-4b18-b769-a7667c20495c","Type":"ContainerStarted","Data":"e91643be4e61edcdc6c7bb39cb4d8ad49bcf07a9efbbad2c6d7d78b92e06ab51"} Sep 18 21:07:24.634004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:24.633982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mb586" event={"ID":"eb0a4e2b-b05e-49fb-b463-7bf6bd793eb6","Type":"ContainerStarted","Data":"f7cbcd25e980303befe9cec9a7e17693a146c228d5cc28e314e46ea810a1e387"} Sep 18 21:07:24.634113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:24.634042 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:07:24.649090 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:24.649039 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6xgxk" podStartSLOduration=67.292413453 podStartE2EDuration="1m11.649023339s" totalImagesPullingTime="4.356609886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:24.647647026 +0000 UTC m=+73.577392900" watchObservedRunningTime="2026-09-18 21:07:24.649023339 +0000 UTC m=+73.578769203" Sep 18 21:07:24.664449 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:24.664386 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mb586" podStartSLOduration=67.327982797 podStartE2EDuration="1m11.664367778s" totalImagesPullingTime="4.336384981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:24.663629834 +0000 UTC m=+73.593375706" watchObservedRunningTime="2026-09-18 21:07:24.664367778 +0000 UTC m=+73.594113652" Sep 18 21:07:25.257981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:25.257920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:25.258425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:25.258002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:25.258425 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:25.258075 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 21:07:25.258425 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:25.258095 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 21:07:25.258425 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:25.258127 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:29.258113538 +0000 UTC m=+78.187859403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : configmap "kube-rbac-proxy" not found Sep 18 21:07:25.258425 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:25.258167 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:29.25814732 +0000 UTC m=+78.187893186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : secret "insights-runtime-extractor-tls" not found Sep 18 21:07:25.321822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:25.321795 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9zlx2_23c22257-3ecd-4264-89ef-560ecd2a6c41/node-ca/0.log" Sep 18 21:07:26.122622 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:26.122589 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dvqc2_f1aca64a-306c-4902-94c5-0f3845ac290b/serve-healthcheck-canary/0.log" Sep 18 21:07:29.288130 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:29.288093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:29.288588 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:29.288147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:29.288588 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:29.288214 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 21:07:29.288588 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:29.288252 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 21:07:29.288588 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:29.288281 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:37.288267463 +0000 UTC m=+86.218013328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : configmap "kube-rbac-proxy" not found Sep 18 21:07:29.288588 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:29.288318 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls podName:f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec nodeName:}" failed. No retries permitted until 2026-09-18 21:07:37.288304429 +0000 UTC m=+86.218050280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqmw5" (UID: "f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec") : secret "insights-runtime-extractor-tls" not found Sep 18 21:07:32.312787 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:32.312748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:32.313235 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:32.312827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:32.313235 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:32.312975 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:32.313235 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:07:32.313044 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls podName:b4311bed-80a0-4146-9ec8-ecd4daedaf0c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:48.313026136 +0000 UTC m=+97.242772003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-kls8k" (UID: "b4311bed-80a0-4146-9ec8-ecd4daedaf0c") : secret "cluster-monitoring-operator-tls" not found Sep 18 21:07:32.315254 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:32.315231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"image-registry-67ff68f94b-j7d5n\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:32.353650 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:32.353610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:32.631895 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:32.631871 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67ff68f94b-j7d5n"] Sep 18 21:07:32.634226 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:32.634198 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6faf6bb_009c_44d1_b93f_197543ed278c.slice/crio-ff7683997b2325d26814b9afcab1f93607b9a9a8bb297f016663526c936329ce WatchSource:0}: Error finding container ff7683997b2325d26814b9afcab1f93607b9a9a8bb297f016663526c936329ce: Status 404 returned error can't find the container with id ff7683997b2325d26814b9afcab1f93607b9a9a8bb297f016663526c936329ce Sep 18 21:07:32.658873 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:32.658837 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" event={"ID":"d6faf6bb-009c-44d1-b93f-197543ed278c","Type":"ContainerStarted","Data":"ff7683997b2325d26814b9afcab1f93607b9a9a8bb297f016663526c936329ce"} Sep 18 21:07:33.662711 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:33.662674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" event={"ID":"d6faf6bb-009c-44d1-b93f-197543ed278c","Type":"ContainerStarted","Data":"7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818"} Sep 18 21:07:33.663168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:33.662810 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:07:33.688365 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:33.688306 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" podStartSLOduration=17.688291092 podStartE2EDuration="17.688291092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:33.686592462 +0000 UTC m=+82.616338338" watchObservedRunningTime="2026-09-18 21:07:33.688291092 +0000 UTC m=+82.618036964" Sep 18 21:07:37.353013 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:37.352976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:37.353420 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:37.353049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:37.353619 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:37.353586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:37.355397 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:37.355363 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqmw5\" (UID: \"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec\") " pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:37.509152 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:37.509118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cqmw5" Sep 18 21:07:37.788944 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:37.788874 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cqmw5"] Sep 18 21:07:37.791889 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:37.791863 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87ef4b1_04e8_4aea_9190_e5a4d0fd74ec.slice/crio-6df8ba1ab96dcc216a9f85af75372c215c1f07e4bbe109c13ba3a098dc9c219b WatchSource:0}: Error finding container 6df8ba1ab96dcc216a9f85af75372c215c1f07e4bbe109c13ba3a098dc9c219b: Status 404 returned error can't find the container with id 6df8ba1ab96dcc216a9f85af75372c215c1f07e4bbe109c13ba3a098dc9c219b Sep 18 21:07:38.682230 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:38.682190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cqmw5" event={"ID":"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec","Type":"ContainerStarted","Data":"938804db11f314adce9431cb974e24c68e394fc0491a9e09e089b1c81c385d43"} Sep 18 21:07:38.682589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:38.682244 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cqmw5" event={"ID":"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec","Type":"ContainerStarted","Data":"7bebdfa535c09ab8df1420e2e7e337efe9dfe04f612e990ce2ce3a7d01ec5ff5"} Sep 18 21:07:38.682589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:38.682263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cqmw5" event={"ID":"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec","Type":"ContainerStarted","Data":"6df8ba1ab96dcc216a9f85af75372c215c1f07e4bbe109c13ba3a098dc9c219b"} Sep 18 21:07:40.689986 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:40.689936 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cqmw5" event={"ID":"f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec","Type":"ContainerStarted","Data":"3861f2563228430ef9018406e3a5ebdaf0ffc64a6d85b8584d0afa55f1fc34d6"} Sep 18 21:07:40.712068 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:40.712024 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cqmw5" podStartSLOduration=17.73683009 podStartE2EDuration="19.712009054s" totalImagesPullingTime="1.975178964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:07:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:40.710882475 +0000 UTC m=+89.640628348" watchObservedRunningTime="2026-09-18 21:07:40.712009054 +0000 UTC m=+89.641754926" Sep 18 21:07:47.436373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.436334 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg"] Sep 18 21:07:47.438302 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.438286 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x"] Sep 18 21:07:47.438427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.438409 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" Sep 18 21:07:47.441161 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.441139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.445842 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.445807 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qmm4x\"" Sep 18 21:07:47.445842 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.445817 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 18 21:07:47.446040 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.445843 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 18 21:07:47.446040 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.445966 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 18 21:07:47.446138 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.446086 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 18 21:07:47.446264 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.446251 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 18 21:07:47.455263 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.455247 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 18 21:07:47.457997 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.457979 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg"] Sep 18 21:07:47.485459 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.485434 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x"] Sep 18 21:07:47.515185 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.515156 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67ff68f94b-j7d5n"] Sep 18 21:07:47.519498 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.519469 2577 patch_prober.go:28] interesting pod/image-registry-67ff68f94b-j7d5n container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 18 21:07:47.519633 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.519529 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" podUID="d6faf6bb-009c-44d1-b93f-197543ed278c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 21:07:47.527685 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.527664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bthxp\" (UniqueName: \"kubernetes.io/projected/4b6b124f-acd2-4cea-b503-d2c0f8805a03-kube-api-access-bthxp\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.527780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.527694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6795d358-7432-4dbc-a986-3bb8d282823b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-tc6hg\" (UID: \"6795d358-7432-4dbc-a986-3bb8d282823b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" Sep 18 21:07:47.527829 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.527781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b6b124f-acd2-4cea-b503-d2c0f8805a03-tmp\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.527829 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.527821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6795d358-7432-4dbc-a986-3bb8d282823b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-tc6hg\" (UID: \"6795d358-7432-4dbc-a986-3bb8d282823b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" Sep 18 21:07:47.527907 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.527863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4b6b124f-acd2-4cea-b503-d2c0f8805a03-klusterlet-config\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.537822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.537799 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc"] Sep 18 21:07:47.539877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.539863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc" Sep 18 21:07:47.543142 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.543124 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2kqhs\"" Sep 18 21:07:47.573883 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.573858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc"] Sep 18 21:07:47.628541 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.628508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6795d358-7432-4dbc-a986-3bb8d282823b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-tc6hg\" (UID: \"6795d358-7432-4dbc-a986-3bb8d282823b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" Sep 18 21:07:47.628541 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.628548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx66s\" (UniqueName: \"kubernetes.io/projected/b7d3f09d-ea37-485e-8a44-11c90b1ade38-kube-api-access-tx66s\") pod \"network-check-source-7cbc45fddc-m7zbc\" (UID: \"b7d3f09d-ea37-485e-8a44-11c90b1ade38\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc" Sep 18 21:07:47.628856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.628665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4b6b124f-acd2-4cea-b503-d2c0f8805a03-klusterlet-config\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.628856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.628726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthxp\" (UniqueName: \"kubernetes.io/projected/4b6b124f-acd2-4cea-b503-d2c0f8805a03-kube-api-access-bthxp\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.628856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.628748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6795d358-7432-4dbc-a986-3bb8d282823b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-tc6hg\" (UID: \"6795d358-7432-4dbc-a986-3bb8d282823b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" Sep 18 21:07:47.628856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.628791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b6b124f-acd2-4cea-b503-d2c0f8805a03-tmp\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.629211 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.629183 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b6b124f-acd2-4cea-b503-d2c0f8805a03-tmp\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.629471 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.629452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6795d358-7432-4dbc-a986-3bb8d282823b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-tc6hg\" (UID: \"6795d358-7432-4dbc-a986-3bb8d282823b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" Sep 18 21:07:47.631003 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.630985 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6795d358-7432-4dbc-a986-3bb8d282823b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-tc6hg\" (UID: \"6795d358-7432-4dbc-a986-3bb8d282823b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" Sep 18 21:07:47.631178 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.631164 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4b6b124f-acd2-4cea-b503-d2c0f8805a03-klusterlet-config\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.696752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.696690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthxp\" (UniqueName: \"kubernetes.io/projected/4b6b124f-acd2-4cea-b503-d2c0f8805a03-kube-api-access-bthxp\") pod \"klusterlet-addon-workmgr-669547585b-2k42x\" (UID: \"4b6b124f-acd2-4cea-b503-d2c0f8805a03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.729457 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.729426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx66s\" (UniqueName: \"kubernetes.io/projected/b7d3f09d-ea37-485e-8a44-11c90b1ade38-kube-api-access-tx66s\") pod \"network-check-source-7cbc45fddc-m7zbc\" (UID: \"b7d3f09d-ea37-485e-8a44-11c90b1ade38\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc" Sep 18 21:07:47.751730 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.751693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" Sep 18 21:07:47.756736 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.756715 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:47.771191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.771164 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx66s\" (UniqueName: \"kubernetes.io/projected/b7d3f09d-ea37-485e-8a44-11c90b1ade38-kube-api-access-tx66s\") pod \"network-check-source-7cbc45fddc-m7zbc\" (UID: \"b7d3f09d-ea37-485e-8a44-11c90b1ade38\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc" Sep 18 21:07:47.814241 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.814207 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-867666fdd5-bbs7d"] Sep 18 21:07:47.817702 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.817680 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:47.842080 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.842053 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-867666fdd5-bbs7d"] Sep 18 21:07:47.847712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.847691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc" Sep 18 21:07:47.930803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.930774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-bound-sa-token\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:47.930951 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.930821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtrl\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-kube-api-access-kgtrl\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:47.930951 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.930898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/423706e6-0050-48a4-afa1-5263ef5cb93b-ca-trust-extracted\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:47.930951 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.930935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-registry-tls\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:47.931076 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.930979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/423706e6-0050-48a4-afa1-5263ef5cb93b-trusted-ca\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:47.931076 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.931031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/423706e6-0050-48a4-afa1-5263ef5cb93b-image-registry-private-configuration\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:47.931076 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.931058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/423706e6-0050-48a4-afa1-5263ef5cb93b-registry-certificates\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:47.931174 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:47.931076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/423706e6-0050-48a4-afa1-5263ef5cb93b-installation-pull-secrets\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.025448 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.025373 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg"] Sep 18 21:07:48.028725 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:48.028702 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6795d358_7432_4dbc_a986_3bb8d282823b.slice/crio-b0253625c6baef6d1e68631dd12a49868df907325e6450273be2893aa4e269ff WatchSource:0}: Error finding container b0253625c6baef6d1e68631dd12a49868df907325e6450273be2893aa4e269ff: Status 404 returned error can't find the container with id b0253625c6baef6d1e68631dd12a49868df907325e6450273be2893aa4e269ff Sep 18 21:07:48.031647 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.031625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtrl\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-kube-api-access-kgtrl\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.031768 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.031666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/423706e6-0050-48a4-afa1-5263ef5cb93b-ca-trust-extracted\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.031768 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.031702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-registry-tls\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.031768 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.031728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/423706e6-0050-48a4-afa1-5263ef5cb93b-trusted-ca\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.031935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.031783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/423706e6-0050-48a4-afa1-5263ef5cb93b-image-registry-private-configuration\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.031935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.031809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/423706e6-0050-48a4-afa1-5263ef5cb93b-registry-certificates\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.031935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.031835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/423706e6-0050-48a4-afa1-5263ef5cb93b-installation-pull-secrets\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.031935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.031893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-bound-sa-token\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.032170 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.032050 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/423706e6-0050-48a4-afa1-5263ef5cb93b-ca-trust-extracted\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.032984 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.032921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/423706e6-0050-48a4-afa1-5263ef5cb93b-registry-certificates\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.033086 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.033066 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x"] Sep 18 21:07:48.033601 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.033503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/423706e6-0050-48a4-afa1-5263ef5cb93b-trusted-ca\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.034680 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.034660 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/423706e6-0050-48a4-afa1-5263ef5cb93b-installation-pull-secrets\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.034856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.034834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-registry-tls\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.035028 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.035013 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/423706e6-0050-48a4-afa1-5263ef5cb93b-image-registry-private-configuration\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.035926 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:48.035866 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6b124f_acd2_4cea_b503_d2c0f8805a03.slice/crio-134e561c9c79780fd0795b7bdb3724cab093b2007351e85cec170f157030bfe9 WatchSource:0}: Error finding container 134e561c9c79780fd0795b7bdb3724cab093b2007351e85cec170f157030bfe9: Status 404 returned error can't find the container with id 134e561c9c79780fd0795b7bdb3724cab093b2007351e85cec170f157030bfe9 Sep 18 21:07:48.053430 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.053401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtrl\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-kube-api-access-kgtrl\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.055341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.055325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/423706e6-0050-48a4-afa1-5263ef5cb93b-bound-sa-token\") pod \"image-registry-867666fdd5-bbs7d\" (UID: \"423706e6-0050-48a4-afa1-5263ef5cb93b\") " pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.127995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.127946 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.129317 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.129283 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc"] Sep 18 21:07:48.132181 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:48.132144 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d3f09d_ea37_485e_8a44_11c90b1ade38.slice/crio-25307ac1934828c6ae4857ddb1e5e77f9386b571c1c9d263aedd633f647f1085 WatchSource:0}: Error finding container 25307ac1934828c6ae4857ddb1e5e77f9386b571c1c9d263aedd633f647f1085: Status 404 returned error can't find the container with id 25307ac1934828c6ae4857ddb1e5e77f9386b571c1c9d263aedd633f647f1085 Sep 18 21:07:48.333692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.333665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:48.336076 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.336048 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4311bed-80a0-4146-9ec8-ecd4daedaf0c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-kls8k\" (UID: \"b4311bed-80a0-4146-9ec8-ecd4daedaf0c\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:48.407894 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.407861 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-867666fdd5-bbs7d"] Sep 18 21:07:48.411101 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:48.411070 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423706e6_0050_48a4_afa1_5263ef5cb93b.slice/crio-44156e78116d6c61c34d67bcb97304e2aceb8df39ee3e19133f73005342f5ee1 WatchSource:0}: Error finding container 44156e78116d6c61c34d67bcb97304e2aceb8df39ee3e19133f73005342f5ee1: Status 404 returned error can't find the container with id 44156e78116d6c61c34d67bcb97304e2aceb8df39ee3e19133f73005342f5ee1 Sep 18 21:07:48.557543 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.557497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" Sep 18 21:07:48.714511 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.714474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" event={"ID":"4b6b124f-acd2-4cea-b503-d2c0f8805a03","Type":"ContainerStarted","Data":"134e561c9c79780fd0795b7bdb3724cab093b2007351e85cec170f157030bfe9"} Sep 18 21:07:48.716164 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.716138 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc" event={"ID":"b7d3f09d-ea37-485e-8a44-11c90b1ade38","Type":"ContainerStarted","Data":"68e86e1bbeef0dcece3c57b40af67a6ab17a30580a75d3f182e049a2d7bcd75b"} Sep 18 21:07:48.716278 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.716185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc" event={"ID":"b7d3f09d-ea37-485e-8a44-11c90b1ade38","Type":"ContainerStarted","Data":"25307ac1934828c6ae4857ddb1e5e77f9386b571c1c9d263aedd633f647f1085"} Sep 18 21:07:48.717795 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.717767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" event={"ID":"423706e6-0050-48a4-afa1-5263ef5cb93b","Type":"ContainerStarted","Data":"2c5f6db49649dfea2dbae4b2157a8270f11eff67fa51e8ffe7c4a9ef03ee0cfc"} Sep 18 21:07:48.717914 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.717817 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:07:48.717914 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.717839 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" event={"ID":"423706e6-0050-48a4-afa1-5263ef5cb93b","Type":"ContainerStarted","Data":"44156e78116d6c61c34d67bcb97304e2aceb8df39ee3e19133f73005342f5ee1"} Sep 18 21:07:48.719056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.719032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" event={"ID":"6795d358-7432-4dbc-a986-3bb8d282823b","Type":"ContainerStarted","Data":"b0253625c6baef6d1e68631dd12a49868df907325e6450273be2893aa4e269ff"} Sep 18 21:07:48.738848 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.738799 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7zbc" podStartSLOduration=1.738780011 podStartE2EDuration="1.738780011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:48.737240786 +0000 UTC m=+97.666986673" watchObservedRunningTime="2026-09-18 21:07:48.738780011 +0000 UTC m=+97.668525886" Sep 18 21:07:48.777290 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.777223 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" podStartSLOduration=1.777201826 podStartE2EDuration="1.777201826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:48.776582405 +0000 UTC m=+97.706328278" watchObservedRunningTime="2026-09-18 21:07:48.777201826 +0000 UTC m=+97.706947686" Sep 18 21:07:48.835393 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:48.834998 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k"] Sep 18 21:07:48.838358 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:48.838327 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4311bed_80a0_4146_9ec8_ecd4daedaf0c.slice/crio-214b27382b43d311142a830be4a74b49f0481969440f1a1d4f5e3a003af2b984 WatchSource:0}: Error finding container 214b27382b43d311142a830be4a74b49f0481969440f1a1d4f5e3a003af2b984: Status 404 returned error can't find the container with id 214b27382b43d311142a830be4a74b49f0481969440f1a1d4f5e3a003af2b984 Sep 18 21:07:49.726226 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:49.726184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" event={"ID":"6795d358-7432-4dbc-a986-3bb8d282823b","Type":"ContainerStarted","Data":"1453625a48c98c7b157c818af96481d4e150ee4946a1d704cea69cf46b597189"} Sep 18 21:07:49.727687 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:49.727662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" event={"ID":"b4311bed-80a0-4146-9ec8-ecd4daedaf0c","Type":"ContainerStarted","Data":"214b27382b43d311142a830be4a74b49f0481969440f1a1d4f5e3a003af2b984"} Sep 18 21:07:49.766195 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:49.766099 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-tc6hg" podStartSLOduration=1.187344679 podStartE2EDuration="2.7660785s" totalImagesPullingTime="1.578733821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:49.764689552 +0000 UTC m=+98.694435427" watchObservedRunningTime="2026-09-18 21:07:49.7660785 +0000 UTC m=+98.695824367" Sep 18 21:07:50.862866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:50.862828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:07:50.865861 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:50.865832 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b085f059-d211-47ed-9747-bb25ea39e37d-metrics-tls\") pod \"dns-default-hpqpv\" (UID: \"b085f059-d211-47ed-9747-bb25ea39e37d\") " pod="openshift-dns/dns-default-hpqpv" Sep 18 21:07:50.898489 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:50.898458 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4xcf\"" Sep 18 21:07:50.905948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:50.905915 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hpqpv" Sep 18 21:07:52.617311 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:52.617284 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hpqpv"] Sep 18 21:07:52.619320 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:07:52.619284 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb085f059_d211_47ed_9747_bb25ea39e37d.slice/crio-fd42516bc6f24bd79a1b422952924a9d796979ccdf23b1c90f81615764de9230 WatchSource:0}: Error finding container fd42516bc6f24bd79a1b422952924a9d796979ccdf23b1c90f81615764de9230: Status 404 returned error can't find the container with id fd42516bc6f24bd79a1b422952924a9d796979ccdf23b1c90f81615764de9230 Sep 18 21:07:52.741539 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:52.741500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" event={"ID":"b4311bed-80a0-4146-9ec8-ecd4daedaf0c","Type":"ContainerStarted","Data":"a26c97ca9200023a3fc342811ab9cd3a7d0d57f992b0f5a8c24c6c6e4e97d3b3"} Sep 18 21:07:52.743134 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:52.743109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" event={"ID":"4b6b124f-acd2-4cea-b503-d2c0f8805a03","Type":"ContainerStarted","Data":"ef1b7cd8d6a036db79476515a3503815c1173f14255192ac723e8d7e1298ff0d"} Sep 18 21:07:52.743316 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:52.743299 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:52.744424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:52.744400 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hpqpv" event={"ID":"b085f059-d211-47ed-9747-bb25ea39e37d","Type":"ContainerStarted","Data":"fd42516bc6f24bd79a1b422952924a9d796979ccdf23b1c90f81615764de9230"} Sep 18 21:07:52.745194 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:52.745172 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" Sep 18 21:07:52.761330 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:52.761276 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-kls8k" podStartSLOduration=33.249369385 podStartE2EDuration="36.761256533s" totalImagesPullingTime="3.511887148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:52.76039275 +0000 UTC m=+101.690138624" watchObservedRunningTime="2026-09-18 21:07:52.761256533 +0000 UTC m=+101.691002461" Sep 18 21:07:52.786676 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:52.786631 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-669547585b-2k42x" podStartSLOduration=1.456989221 podStartE2EDuration="5.786615526s" totalImagesPullingTime="4.329626305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:52.785592922 +0000 UTC m=+101.715338794" watchObservedRunningTime="2026-09-18 21:07:52.786615526 +0000 UTC m=+101.716361399" Sep 18 21:07:55.639220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:55.639185 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mb586" Sep 18 21:07:55.756479 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:55.756443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hpqpv" event={"ID":"b085f059-d211-47ed-9747-bb25ea39e37d","Type":"ContainerStarted","Data":"77db0d00fc269d071547bd27cb688562fee4f5f8ef5b0b4abf6f0d34fcbb019e"} Sep 18 21:07:55.756693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:55.756493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hpqpv" event={"ID":"b085f059-d211-47ed-9747-bb25ea39e37d","Type":"ContainerStarted","Data":"d5307dc8baa46282fd387762c0530be1bfc46d6ccc4ee7d4511e738849ebfd9c"} Sep 18 21:07:55.756693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:55.756570 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hpqpv" Sep 18 21:07:55.798069 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:55.798013 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hpqpv" podStartSLOduration=67.616119141 podStartE2EDuration="1m9.797997016s" totalImagesPullingTime="2.181877875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:55.797450474 +0000 UTC m=+104.727196347" watchObservedRunningTime="2026-09-18 21:07:55.797997016 +0000 UTC m=+104.727742890" Sep 18 21:07:57.519635 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:57.519602 2577 patch_prober.go:28] interesting pod/image-registry-67ff68f94b-j7d5n container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 18 21:07:57.520098 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:57.519674 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" podUID="d6faf6bb-009c-44d1-b93f-197543ed278c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 21:07:59.608748 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.608713 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-964796757-hlccj"] Sep 18 21:07:59.610769 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.610752 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.614682 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.614662 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 18 21:07:59.615782 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.615762 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 21:07:59.615893 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.615796 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 18 21:07:59.616006 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.615833 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 21:07:59.616087 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.615850 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9pwwm\"" Sep 18 21:07:59.616654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.616637 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 21:07:59.616750 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.616727 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 21:07:59.616825 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.616801 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 21:07:59.621340 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.621322 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 18 21:07:59.626544 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.626522 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-964796757-hlccj"] Sep 18 21:07:59.737242 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.737191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-oauth-config\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.737242 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.737242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-config\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.737457 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.737278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-oauth-serving-cert\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.737457 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.737360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-trusted-ca-bundle\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.737457 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.737394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-kube-api-access-n9m5k\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.737567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.737460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-serving-cert\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.737567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.737485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-service-ca\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.838466 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.838418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-serving-cert\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.838466 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.838457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-service-ca\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.838717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.838482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-oauth-config\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.838717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.838513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-config\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.838717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.838559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-oauth-serving-cert\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.838717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.838606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-trusted-ca-bundle\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.838717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.838639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-kube-api-access-n9m5k\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.839321 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.839284 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-service-ca\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.839431 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.839378 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-config\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.839431 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.839395 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-oauth-serving-cert\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.839536 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.839520 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-trusted-ca-bundle\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.841522 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.841501 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-oauth-config\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.841591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.841577 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-serving-cert\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.848817 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.848799 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-kube-api-access-n9m5k\") pod \"console-964796757-hlccj\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " pod="openshift-console/console-964796757-hlccj" Sep 18 21:07:59.920555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:07:59.920464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-964796757-hlccj" Sep 18 21:08:00.193368 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:00.193276 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-964796757-hlccj"] Sep 18 21:08:00.196486 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:00.196459 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd57065c_a1b2_4e3f_aba6_d897f8dac6db.slice/crio-fcf148fa210ea1082c535c940d909ef542b3d23fd19f60ba8ba942dc9cc13761 WatchSource:0}: Error finding container fcf148fa210ea1082c535c940d909ef542b3d23fd19f60ba8ba942dc9cc13761: Status 404 returned error can't find the container with id fcf148fa210ea1082c535c940d909ef542b3d23fd19f60ba8ba942dc9cc13761 Sep 18 21:08:00.774802 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:00.774757 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-964796757-hlccj" event={"ID":"bd57065c-a1b2-4e3f-aba6-d897f8dac6db","Type":"ContainerStarted","Data":"fcf148fa210ea1082c535c940d909ef542b3d23fd19f60ba8ba942dc9cc13761"} Sep 18 21:08:04.791368 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:04.791334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-964796757-hlccj" event={"ID":"bd57065c-a1b2-4e3f-aba6-d897f8dac6db","Type":"ContainerStarted","Data":"751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3"} Sep 18 21:08:04.824435 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:04.824385 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-964796757-hlccj" podStartSLOduration=2.000616677 podStartE2EDuration="5.824368171s" totalImagesPullingTime="3.823751494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:04.822572913 +0000 UTC m=+113.752318786" watchObservedRunningTime="2026-09-18 21:08:04.824368171 +0000 UTC m=+113.754114041" Sep 18 21:08:05.761348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:05.761317 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hpqpv" Sep 18 21:08:06.101795 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.101757 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w"] Sep 18 21:08:06.104311 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.104295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.111267 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.111251 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 18 21:08:06.111347 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.111279 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 18 21:08:06.112321 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.112302 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6f7cc\"" Sep 18 21:08:06.112398 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.112304 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 21:08:06.134854 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.134829 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w"] Sep 18 21:08:06.297747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.297711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.297943 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.297768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.297943 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.297793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8dr\" (UniqueName: \"kubernetes.io/projected/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-kube-api-access-kf8dr\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.297943 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.297863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.398647 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.398549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.398647 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.398605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.398647 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.398628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8dr\" (UniqueName: \"kubernetes.io/projected/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-kube-api-access-kf8dr\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.398928 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.398674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.399259 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.399216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.401143 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.401114 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.401143 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.401134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.418721 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.418693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8dr\" (UniqueName: \"kubernetes.io/projected/bb7a7257-28fa-483c-9cce-0a00bc5c0f66-kube-api-access-kf8dr\") pod \"prometheus-operator-7bd5d56d6-sqn2w\" (UID: \"bb7a7257-28fa-483c-9cce-0a00bc5c0f66\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:06.714014 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:06.713899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" Sep 18 21:08:07.010600 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:07.010513 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w"] Sep 18 21:08:07.014966 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:07.014925 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7a7257_28fa_483c_9cce_0a00bc5c0f66.slice/crio-daff1209a694472159bc33bcf93ed46c9a396721b2da1f61415d744746875dbc WatchSource:0}: Error finding container daff1209a694472159bc33bcf93ed46c9a396721b2da1f61415d744746875dbc: Status 404 returned error can't find the container with id daff1209a694472159bc33bcf93ed46c9a396721b2da1f61415d744746875dbc Sep 18 21:08:07.520350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:07.520327 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:08:07.801279 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:07.801190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" event={"ID":"bb7a7257-28fa-483c-9cce-0a00bc5c0f66","Type":"ContainerStarted","Data":"daff1209a694472159bc33bcf93ed46c9a396721b2da1f61415d744746875dbc"} Sep 18 21:08:08.806693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:08.806616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" event={"ID":"bb7a7257-28fa-483c-9cce-0a00bc5c0f66","Type":"ContainerStarted","Data":"4998de1ee59860266b5db0c7fe6038bc9e98c211d89c7e8471f196ebc3a27dee"} Sep 18 21:08:08.806693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:08.806665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" event={"ID":"bb7a7257-28fa-483c-9cce-0a00bc5c0f66","Type":"ContainerStarted","Data":"eb7f2bb6353d67fb907978515deeaa8ddee4030ee5566e074e271fab4d1f4319"} Sep 18 21:08:08.827228 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:08.827182 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-sqn2w" podStartSLOduration=1.4391150449999999 podStartE2EDuration="2.827167156s" totalImagesPullingTime="1.388052111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:08.826092261 +0000 UTC m=+117.755838133" watchObservedRunningTime="2026-09-18 21:08:08.827167156 +0000 UTC m=+117.756913029" Sep 18 21:08:09.732000 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:09.731973 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-867666fdd5-bbs7d" Sep 18 21:08:09.921518 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:09.921483 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-964796757-hlccj" Sep 18 21:08:09.921902 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:09.921558 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-964796757-hlccj" Sep 18 21:08:09.926244 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:09.926225 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-964796757-hlccj" Sep 18 21:08:10.519597 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.519568 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq"] Sep 18 21:08:10.521606 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.521591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.524759 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.524742 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 18 21:08:10.525205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.525185 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 18 21:08:10.525277 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.525248 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pdszh\"" Sep 18 21:08:10.534758 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.534738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/260c0408-1d9c-45c8-93ef-54f75c774bfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.534845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.534768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/260c0408-1d9c-45c8-93ef-54f75c774bfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.534845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.534806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r988f\" (UniqueName: \"kubernetes.io/projected/260c0408-1d9c-45c8-93ef-54f75c774bfc-kube-api-access-r988f\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.534945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.534862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/260c0408-1d9c-45c8-93ef-54f75c774bfc-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.550941 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.550918 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq"] Sep 18 21:08:10.561762 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.561731 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2hwfk"] Sep 18 21:08:10.563943 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.563925 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.566403 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.566383 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 21:08:10.566510 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.566470 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 21:08:10.566510 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.566494 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cktl2\"" Sep 18 21:08:10.566628 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.566512 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 21:08:10.636100 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/842cb176-3c0b-4588-8409-03422d790d40-metrics-client-ca\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636100 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-tls\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-sys\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-root\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-wtmp\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/260c0408-1d9c-45c8-93ef-54f75c774bfc-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.636472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-accelerators-collector-config\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/260c0408-1d9c-45c8-93ef-54f75c774bfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.636472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/260c0408-1d9c-45c8-93ef-54f75c774bfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.636472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636422 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-textfile\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rx8n\" (UniqueName: \"kubernetes.io/projected/842cb176-3c0b-4588-8409-03422d790d40-kube-api-access-4rx8n\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.636626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r988f\" (UniqueName: \"kubernetes.io/projected/260c0408-1d9c-45c8-93ef-54f75c774bfc-kube-api-access-r988f\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.636979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.636939 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/260c0408-1d9c-45c8-93ef-54f75c774bfc-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.638689 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.638667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/260c0408-1d9c-45c8-93ef-54f75c774bfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.638789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.638773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/260c0408-1d9c-45c8-93ef-54f75c774bfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.652044 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.652018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r988f\" (UniqueName: \"kubernetes.io/projected/260c0408-1d9c-45c8-93ef-54f75c774bfc-kube-api-access-r988f\") pod \"openshift-state-metrics-86c5bc79d6-n66rq\" (UID: \"260c0408-1d9c-45c8-93ef-54f75c774bfc\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.737341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-textfile\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rx8n\" (UniqueName: \"kubernetes.io/projected/842cb176-3c0b-4588-8409-03422d790d40-kube-api-access-4rx8n\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/842cb176-3c0b-4588-8409-03422d790d40-metrics-client-ca\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-tls\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-sys\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-root\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-wtmp\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-accelerators-collector-config\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737562 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-sys\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-root\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-textfile\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.737888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.737737 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-wtmp\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.738125 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.738101 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-accelerators-collector-config\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.738188 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.738108 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/842cb176-3c0b-4588-8409-03422d790d40-metrics-client-ca\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.739786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.739758 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.739876 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.739823 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/842cb176-3c0b-4588-8409-03422d790d40-node-exporter-tls\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.747774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.747750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rx8n\" (UniqueName: \"kubernetes.io/projected/842cb176-3c0b-4588-8409-03422d790d40-kube-api-access-4rx8n\") pod \"node-exporter-2hwfk\" (UID: \"842cb176-3c0b-4588-8409-03422d790d40\") " pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.818536 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.818511 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-964796757-hlccj" Sep 18 21:08:10.830860 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.830839 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" Sep 18 21:08:10.873460 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:10.873425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2hwfk" Sep 18 21:08:10.886008 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:10.885972 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842cb176_3c0b_4588_8409_03422d790d40.slice/crio-599de36ee444a227a9c44887436b33c372a743e7c5e7398b36c6a00fd579f41c WatchSource:0}: Error finding container 599de36ee444a227a9c44887436b33c372a743e7c5e7398b36c6a00fd579f41c: Status 404 returned error can't find the container with id 599de36ee444a227a9c44887436b33c372a743e7c5e7398b36c6a00fd579f41c Sep 18 21:08:11.103167 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:11.102975 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq"] Sep 18 21:08:11.105579 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:11.105550 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260c0408_1d9c_45c8_93ef_54f75c774bfc.slice/crio-7c91bb0900e22b30b1e81774da0d489f39bfc63ea2545dcc241a001d4b6aeeb6 WatchSource:0}: Error finding container 7c91bb0900e22b30b1e81774da0d489f39bfc63ea2545dcc241a001d4b6aeeb6: Status 404 returned error can't find the container with id 7c91bb0900e22b30b1e81774da0d489f39bfc63ea2545dcc241a001d4b6aeeb6 Sep 18 21:08:11.820418 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:11.820376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2hwfk" event={"ID":"842cb176-3c0b-4588-8409-03422d790d40","Type":"ContainerStarted","Data":"599de36ee444a227a9c44887436b33c372a743e7c5e7398b36c6a00fd579f41c"} Sep 18 21:08:11.822995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:11.822943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" event={"ID":"260c0408-1d9c-45c8-93ef-54f75c774bfc","Type":"ContainerStarted","Data":"98546e6bec4e802abd608d289c6cd7814780fa4b9b7d7d669953d66f4267c7ef"} Sep 18 21:08:11.823136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:11.823029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" event={"ID":"260c0408-1d9c-45c8-93ef-54f75c774bfc","Type":"ContainerStarted","Data":"66f0b2ec844c20fdcc2fd01d927b27dabeae22e5faacb598f780c41efcb8e73a"} Sep 18 21:08:11.823136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:11.823058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" event={"ID":"260c0408-1d9c-45c8-93ef-54f75c774bfc","Type":"ContainerStarted","Data":"7c91bb0900e22b30b1e81774da0d489f39bfc63ea2545dcc241a001d4b6aeeb6"} Sep 18 21:08:12.534926 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.534830 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" podUID="d6faf6bb-009c-44d1-b93f-197543ed278c" containerName="registry" containerID="cri-o://7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818" gracePeriod=30 Sep 18 21:08:12.636666 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.636630 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-97fff58fc-lmwhq"] Sep 18 21:08:12.639195 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.639171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.642901 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.642874 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 18 21:08:12.643218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.643203 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 18 21:08:12.643436 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.643415 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fivvu27uk262i\"" Sep 18 21:08:12.643532 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.643465 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 18 21:08:12.643532 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.643471 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qtq44\"" Sep 18 21:08:12.643532 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.643522 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 18 21:08:12.643688 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.643612 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 18 21:08:12.656746 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.656718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.656844 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.656767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-metrics-client-ca\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.656844 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.656798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.656844 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.656826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv4v7\" (UniqueName: \"kubernetes.io/projected/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-kube-api-access-bv4v7\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.656982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.656853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-grpc-tls\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.656982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.656875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.656982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.656932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.656982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.656949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-tls\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.665028 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.665001 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-97fff58fc-lmwhq"] Sep 18 21:08:12.758002 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.757977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-grpc-tls\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.758114 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.758031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.758114 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.758073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.758114 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.758098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-tls\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.758249 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.758148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.758350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.758326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-metrics-client-ca\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.758411 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.758392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.758466 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.758422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv4v7\" (UniqueName: \"kubernetes.io/projected/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-kube-api-access-bv4v7\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.759160 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.759138 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-metrics-client-ca\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.761170 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.761054 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.761170 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.761126 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.761170 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.761158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-tls\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.761342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.761250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.761342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.761319 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-grpc-tls\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.762779 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.762753 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.766832 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.766810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv4v7\" (UniqueName: \"kubernetes.io/projected/ffc465a4-1baf-41fc-8d98-7f81323c3dc6-kube-api-access-bv4v7\") pod \"thanos-querier-97fff58fc-lmwhq\" (UID: \"ffc465a4-1baf-41fc-8d98-7f81323c3dc6\") " pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.784263 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.784241 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:08:12.828029 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.827996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" event={"ID":"260c0408-1d9c-45c8-93ef-54f75c774bfc","Type":"ContainerStarted","Data":"a7b7a3246eeae2d8c4b9f5afed15ba15c09816152e745d008cbcca1f6fd79c73"} Sep 18 21:08:12.829629 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.829607 2577 generic.go:356] "Generic (PLEG): container finished" podID="842cb176-3c0b-4588-8409-03422d790d40" containerID="9f5f4811220d9b892ace8db6de443e4234bf44e7cc7b399b70d5124ff3f72472" exitCode=0 Sep 18 21:08:12.829756 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.829696 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2hwfk" event={"ID":"842cb176-3c0b-4588-8409-03422d790d40","Type":"ContainerDied","Data":"9f5f4811220d9b892ace8db6de443e4234bf44e7cc7b399b70d5124ff3f72472"} Sep 18 21:08:12.831009 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.830987 2577 generic.go:356] "Generic (PLEG): container finished" podID="d6faf6bb-009c-44d1-b93f-197543ed278c" containerID="7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818" exitCode=0 Sep 18 21:08:12.831101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.831031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" event={"ID":"d6faf6bb-009c-44d1-b93f-197543ed278c","Type":"ContainerDied","Data":"7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818"} Sep 18 21:08:12.831101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.831041 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" Sep 18 21:08:12.831101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.831073 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ff68f94b-j7d5n" event={"ID":"d6faf6bb-009c-44d1-b93f-197543ed278c","Type":"ContainerDied","Data":"ff7683997b2325d26814b9afcab1f93607b9a9a8bb297f016663526c936329ce"} Sep 18 21:08:12.831101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.831100 2577 scope.go:117] "RemoveContainer" containerID="7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818" Sep 18 21:08:12.838925 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.838908 2577 scope.go:117] "RemoveContainer" containerID="7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818" Sep 18 21:08:12.839244 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:08:12.839219 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818\": container with ID starting with 7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818 not found: ID does not exist" containerID="7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818" Sep 18 21:08:12.839312 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.839252 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818"} err="failed to get container status \"7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818\": rpc error: code = NotFound desc = could not find container \"7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818\": container with ID starting with 7020987c4a511ac7deff4b985cbba9df5e13d467e0f91900f8f85100c38cf818 not found: ID does not exist" Sep 18 21:08:12.847876 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.847662 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-n66rq" podStartSLOduration=1.81363272 podStartE2EDuration="2.84764612s" totalImagesPullingTime="1.0340134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:12.847418525 +0000 UTC m=+121.777164399" watchObservedRunningTime="2026-09-18 21:08:12.84764612 +0000 UTC m=+121.777391994" Sep 18 21:08:12.859427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859410 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-trusted-ca\") pod \"d6faf6bb-009c-44d1-b93f-197543ed278c\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " Sep 18 21:08:12.859497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859444 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-bound-sa-token\") pod \"d6faf6bb-009c-44d1-b93f-197543ed278c\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " Sep 18 21:08:12.859497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859478 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-installation-pull-secrets\") pod \"d6faf6bb-009c-44d1-b93f-197543ed278c\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " Sep 18 21:08:12.859591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859498 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-image-registry-private-configuration\") pod \"d6faf6bb-009c-44d1-b93f-197543ed278c\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " Sep 18 21:08:12.859591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859525 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr7bg\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-kube-api-access-gr7bg\") pod \"d6faf6bb-009c-44d1-b93f-197543ed278c\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " Sep 18 21:08:12.859591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859547 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") pod \"d6faf6bb-009c-44d1-b93f-197543ed278c\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " Sep 18 21:08:12.859591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859583 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6faf6bb-009c-44d1-b93f-197543ed278c-ca-trust-extracted\") pod \"d6faf6bb-009c-44d1-b93f-197543ed278c\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " Sep 18 21:08:12.860120 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859622 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-certificates\") pod \"d6faf6bb-009c-44d1-b93f-197543ed278c\" (UID: \"d6faf6bb-009c-44d1-b93f-197543ed278c\") " Sep 18 21:08:12.860120 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.859943 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d6faf6bb-009c-44d1-b93f-197543ed278c" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:12.860120 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.860042 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-trusted-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:12.861051 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.861023 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d6faf6bb-009c-44d1-b93f-197543ed278c" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:12.862163 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.862121 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d6faf6bb-009c-44d1-b93f-197543ed278c" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:08:12.862257 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.862209 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-kube-api-access-gr7bg" (OuterVolumeSpecName: "kube-api-access-gr7bg") pod "d6faf6bb-009c-44d1-b93f-197543ed278c" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c"). InnerVolumeSpecName "kube-api-access-gr7bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:08:12.862313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.862249 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d6faf6bb-009c-44d1-b93f-197543ed278c" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:08:12.862439 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.862411 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d6faf6bb-009c-44d1-b93f-197543ed278c" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:08:12.862656 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.862638 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d6faf6bb-009c-44d1-b93f-197543ed278c" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:08:12.868859 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.868832 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6faf6bb-009c-44d1-b93f-197543ed278c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d6faf6bb-009c-44d1-b93f-197543ed278c" (UID: "d6faf6bb-009c-44d1-b93f-197543ed278c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:08:12.948895 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.948874 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:12.961321 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.961299 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-bound-sa-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:12.961412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.961325 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-installation-pull-secrets\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:12.961412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.961339 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6faf6bb-009c-44d1-b93f-197543ed278c-image-registry-private-configuration\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:12.961412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.961355 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr7bg\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-kube-api-access-gr7bg\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:12.961412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.961368 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-tls\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:12.961412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.961380 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6faf6bb-009c-44d1-b93f-197543ed278c-ca-trust-extracted\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:12.961412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:12.961392 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6faf6bb-009c-44d1-b93f-197543ed278c-registry-certificates\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:13.153239 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:13.153203 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67ff68f94b-j7d5n"] Sep 18 21:08:13.157446 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:13.157423 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67ff68f94b-j7d5n"] Sep 18 21:08:13.234422 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:13.234390 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-97fff58fc-lmwhq"] Sep 18 21:08:13.238064 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:13.238034 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc465a4_1baf_41fc_8d98_7f81323c3dc6.slice/crio-e2061d63974d9ba2970c0a29fb189acebc873d81c0980b7a5fa3d7b1cb14cd13 WatchSource:0}: Error finding container e2061d63974d9ba2970c0a29fb189acebc873d81c0980b7a5fa3d7b1cb14cd13: Status 404 returned error can't find the container with id e2061d63974d9ba2970c0a29fb189acebc873d81c0980b7a5fa3d7b1cb14cd13 Sep 18 21:08:13.357319 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:13.357277 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6faf6bb-009c-44d1-b93f-197543ed278c" path="/var/lib/kubelet/pods/d6faf6bb-009c-44d1-b93f-197543ed278c/volumes" Sep 18 21:08:13.836462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:13.836424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2hwfk" event={"ID":"842cb176-3c0b-4588-8409-03422d790d40","Type":"ContainerStarted","Data":"00d87467d1fb25c8590db661988a9bf9a310b37592de2b4300638833debb8a6b"} Sep 18 21:08:13.836871 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:13.836482 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2hwfk" event={"ID":"842cb176-3c0b-4588-8409-03422d790d40","Type":"ContainerStarted","Data":"9f68e0add502ac9e22c8ce3cbb619993cdd7f7d36c7399c716781514375bea21"} Sep 18 21:08:13.837725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:13.837695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" event={"ID":"ffc465a4-1baf-41fc-8d98-7f81323c3dc6","Type":"ContainerStarted","Data":"e2061d63974d9ba2970c0a29fb189acebc873d81c0980b7a5fa3d7b1cb14cd13"} Sep 18 21:08:13.860817 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:13.860772 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2hwfk" podStartSLOduration=2.750752495 podStartE2EDuration="3.860759193s" totalImagesPullingTime="1.03934562s" totalInitContainerRuntime="70.661078ms" isStatefulPod=true podCreationTimestamp="2026-09-18 21:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:13.860333805 +0000 UTC m=+122.790079677" watchObservedRunningTime="2026-09-18 21:08:13.860759193 +0000 UTC m=+122.790505065" Sep 18 21:08:15.343455 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.343429 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d68c4cd5d-th5b8"] Sep 18 21:08:15.343811 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.343700 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6faf6bb-009c-44d1-b93f-197543ed278c" containerName="registry" Sep 18 21:08:15.343811 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.343711 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6faf6bb-009c-44d1-b93f-197543ed278c" containerName="registry" Sep 18 21:08:15.343811 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.343761 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6faf6bb-009c-44d1-b93f-197543ed278c" containerName="registry" Sep 18 21:08:15.347738 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.347718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.361594 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.361571 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-hl85b"] Sep 18 21:08:15.364815 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.364789 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:15.368725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.368677 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 18 21:08:15.369343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.369239 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d68c4cd5d-th5b8"] Sep 18 21:08:15.369343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.369246 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m2l6f\"" Sep 18 21:08:15.384102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.384067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-config\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.384217 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.384160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-service-ca\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.384285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.384222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6spr\" (UniqueName: \"kubernetes.io/projected/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-kube-api-access-v6spr\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.384285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.384253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-serving-cert\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.384390 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.384288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-oauth-config\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.384390 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.384319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-trusted-ca-bundle\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.384390 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.384354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-oauth-serving-cert\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.386472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.386407 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-hl85b"] Sep 18 21:08:15.485117 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-config\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.485309 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-service-ca\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.485309 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znn2v\" (UniqueName: \"kubernetes.io/projected/a8e27e6c-4674-4a31-9898-340876416861-kube-api-access-znn2v\") pod \"monitoring-plugin-5984459ff8-hl85b\" (UID: \"a8e27e6c-4674-4a31-9898-340876416861\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:15.485309 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6spr\" (UniqueName: \"kubernetes.io/projected/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-kube-api-access-v6spr\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.485478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-serving-cert\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.485478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-oauth-config\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.485478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-trusted-ca-bundle\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.485478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8e27e6c-4674-4a31-9898-340876416861-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-hl85b\" (UID: \"a8e27e6c-4674-4a31-9898-340876416861\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:15.485478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-oauth-serving-cert\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.485821 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485799 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-service-ca\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.485998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.485949 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-config\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.486156 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.486137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-oauth-serving-cert\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.486434 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.486417 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-trusted-ca-bundle\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.487897 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.487867 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-oauth-config\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.488008 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.487921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-serving-cert\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.495348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.495295 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6spr\" (UniqueName: \"kubernetes.io/projected/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-kube-api-access-v6spr\") pod \"console-d68c4cd5d-th5b8\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.585934 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.585891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8e27e6c-4674-4a31-9898-340876416861-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-hl85b\" (UID: \"a8e27e6c-4674-4a31-9898-340876416861\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:15.586133 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.586028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znn2v\" (UniqueName: \"kubernetes.io/projected/a8e27e6c-4674-4a31-9898-340876416861-kube-api-access-znn2v\") pod \"monitoring-plugin-5984459ff8-hl85b\" (UID: \"a8e27e6c-4674-4a31-9898-340876416861\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:15.588345 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.588325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8e27e6c-4674-4a31-9898-340876416861-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-hl85b\" (UID: \"a8e27e6c-4674-4a31-9898-340876416861\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:15.594174 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.594154 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znn2v\" (UniqueName: \"kubernetes.io/projected/a8e27e6c-4674-4a31-9898-340876416861-kube-api-access-znn2v\") pod \"monitoring-plugin-5984459ff8-hl85b\" (UID: \"a8e27e6c-4674-4a31-9898-340876416861\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:15.657858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.657815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:15.675262 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.675240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:15.849023 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.848985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" event={"ID":"ffc465a4-1baf-41fc-8d98-7f81323c3dc6","Type":"ContainerStarted","Data":"5e9572405033fedb6f6dd70b213c7c3ddca0cdabeafe5a15cdcae81ad1e034bb"} Sep 18 21:08:15.849191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.849058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" event={"ID":"ffc465a4-1baf-41fc-8d98-7f81323c3dc6","Type":"ContainerStarted","Data":"130e65d17156967cfce8fe98a0b869a2607e18be9e9c01fde8c5704a1de5c9a5"} Sep 18 21:08:15.849191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.849085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" event={"ID":"ffc465a4-1baf-41fc-8d98-7f81323c3dc6","Type":"ContainerStarted","Data":"d47e37896d47dd221ca0e16975b29a1fc61c05922ef1979f2ce47cc70f80bd8c"} Sep 18 21:08:15.963485 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.963446 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d68c4cd5d-th5b8"] Sep 18 21:08:15.969995 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:15.969969 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67ca8e9_bd48_43ed_bb34_c25059cfc36d.slice/crio-d4443aa4c5c25b3184dc8350c27580795b41c821575a34b8408a35429f225038 WatchSource:0}: Error finding container d4443aa4c5c25b3184dc8350c27580795b41c821575a34b8408a35429f225038: Status 404 returned error can't find the container with id d4443aa4c5c25b3184dc8350c27580795b41c821575a34b8408a35429f225038 Sep 18 21:08:15.985341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:15.985313 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-hl85b"] Sep 18 21:08:15.988403 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:15.988388 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e27e6c_4674_4a31_9898_340876416861.slice/crio-9d40979b7d951687dd113f1eefd59a24661d6c872ef9c860c0883a939935a8db WatchSource:0}: Error finding container 9d40979b7d951687dd113f1eefd59a24661d6c872ef9c860c0883a939935a8db: Status 404 returned error can't find the container with id 9d40979b7d951687dd113f1eefd59a24661d6c872ef9c860c0883a939935a8db Sep 18 21:08:16.855038 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.854993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d68c4cd5d-th5b8" event={"ID":"d67ca8e9-bd48-43ed-bb34-c25059cfc36d","Type":"ContainerStarted","Data":"7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089"} Sep 18 21:08:16.855487 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.855058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d68c4cd5d-th5b8" event={"ID":"d67ca8e9-bd48-43ed-bb34-c25059cfc36d","Type":"ContainerStarted","Data":"d4443aa4c5c25b3184dc8350c27580795b41c821575a34b8408a35429f225038"} Sep 18 21:08:16.858225 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.858194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" event={"ID":"a8e27e6c-4674-4a31-9898-340876416861","Type":"ContainerStarted","Data":"9d40979b7d951687dd113f1eefd59a24661d6c872ef9c860c0883a939935a8db"} Sep 18 21:08:16.861471 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.861447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" event={"ID":"ffc465a4-1baf-41fc-8d98-7f81323c3dc6","Type":"ContainerStarted","Data":"de3372b7b406a3ac8d211eb08ea7abd92cf5cee87790e311ddd4d12885193d32"} Sep 18 21:08:16.861601 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.861501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" event={"ID":"ffc465a4-1baf-41fc-8d98-7f81323c3dc6","Type":"ContainerStarted","Data":"86bedfcde58d1f972b3ddc9eb949f419494d2bf96ea8a1418841f3497d15d0e5"} Sep 18 21:08:16.861601 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.861526 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" event={"ID":"ffc465a4-1baf-41fc-8d98-7f81323c3dc6","Type":"ContainerStarted","Data":"dbb2af3256119addc339ac24989ffdee26597d08d882656cf0d665167e2b8f00"} Sep 18 21:08:16.861601 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.861555 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:16.897987 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.897493 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d68c4cd5d-th5b8" podStartSLOduration=1.89747526 podStartE2EDuration="1.89747526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:16.896459142 +0000 UTC m=+125.826205015" watchObservedRunningTime="2026-09-18 21:08:16.89747526 +0000 UTC m=+125.827221133" Sep 18 21:08:16.942413 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:16.942350 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" podStartSLOduration=2.00408259 podStartE2EDuration="4.942328839s" totalImagesPullingTime="2.938246249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:16.94059638 +0000 UTC m=+125.870342253" watchObservedRunningTime="2026-09-18 21:08:16.942328839 +0000 UTC m=+125.872074713" Sep 18 21:08:17.866942 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:17.866902 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" event={"ID":"a8e27e6c-4674-4a31-9898-340876416861","Type":"ContainerStarted","Data":"c1a55872001d068703400f9a3c41ac8e2aab80617be0dc57ee002e7fefafa9d0"} Sep 18 21:08:17.867428 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:17.867310 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:17.872191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:17.872164 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" Sep 18 21:08:17.889374 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:17.887012 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-hl85b" podStartSLOduration=1.449178644 podStartE2EDuration="2.886993132s" totalImagesPullingTime="1.437814488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:17.884576583 +0000 UTC m=+126.814322457" watchObservedRunningTime="2026-09-18 21:08:17.886993132 +0000 UTC m=+126.816739005" Sep 18 21:08:20.872185 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:20.872151 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d68c4cd5d-th5b8"] Sep 18 21:08:20.927917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:20.927878 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bc6795779-cszfk"] Sep 18 21:08:20.932321 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:20.932298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:20.958607 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:20.958572 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc6795779-cszfk"] Sep 18 21:08:21.033686 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.033637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-oauth-config\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.033882 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.033705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-serving-cert\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.033882 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.033777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-trusted-ca-bundle\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.033882 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.033802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzrr\" (UniqueName: \"kubernetes.io/projected/7b8e79c8-ff79-4deb-a577-0a273da5bb13-kube-api-access-gpzrr\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.034063 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.033897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-oauth-serving-cert\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.034063 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.033948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-config\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.034063 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.033999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-service-ca\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.135317 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.135233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-oauth-serving-cert\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.135317 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.135275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-config\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.135317 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.135294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-service-ca\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.135572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.135325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-oauth-config\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.135572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.135360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-serving-cert\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.135572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.135423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-trusted-ca-bundle\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.135572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.135446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzrr\" (UniqueName: \"kubernetes.io/projected/7b8e79c8-ff79-4deb-a577-0a273da5bb13-kube-api-access-gpzrr\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.136164 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.136137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-oauth-serving-cert\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.136352 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.136315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-trusted-ca-bundle\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.136352 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.136162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-config\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.136527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.136152 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-service-ca\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.137939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.137913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-oauth-config\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.137939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.137929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-serving-cert\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.146186 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.146159 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzrr\" (UniqueName: \"kubernetes.io/projected/7b8e79c8-ff79-4deb-a577-0a273da5bb13-kube-api-access-gpzrr\") pod \"console-6bc6795779-cszfk\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.241275 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.241239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:21.545861 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.545833 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc6795779-cszfk"] Sep 18 21:08:21.547566 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:21.547538 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8e79c8_ff79_4deb_a577_0a273da5bb13.slice/crio-e64b9cd2b7f868f37d0621dcb824e64a60c781aafc8946ac507fb777c544251e WatchSource:0}: Error finding container e64b9cd2b7f868f37d0621dcb824e64a60c781aafc8946ac507fb777c544251e: Status 404 returned error can't find the container with id e64b9cd2b7f868f37d0621dcb824e64a60c781aafc8946ac507fb777c544251e Sep 18 21:08:21.885675 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.885635 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc6795779-cszfk" event={"ID":"7b8e79c8-ff79-4deb-a577-0a273da5bb13","Type":"ContainerStarted","Data":"d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562"} Sep 18 21:08:21.886085 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.885697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc6795779-cszfk" event={"ID":"7b8e79c8-ff79-4deb-a577-0a273da5bb13","Type":"ContainerStarted","Data":"e64b9cd2b7f868f37d0621dcb824e64a60c781aafc8946ac507fb777c544251e"} Sep 18 21:08:21.912204 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:21.912151 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bc6795779-cszfk" podStartSLOduration=1.912130405 podStartE2EDuration="1.912130405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:21.910163984 +0000 UTC m=+130.839909857" watchObservedRunningTime="2026-09-18 21:08:21.912130405 +0000 UTC m=+130.841876279" Sep 18 21:08:22.872752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:22.872724 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-97fff58fc-lmwhq" Sep 18 21:08:23.053432 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:23.053392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:08:23.055810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:23.055781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e35eaad-1b7d-40d2-a280-ef6d3c538a8c-metrics-certs\") pod \"network-metrics-daemon-rt75r\" (UID: \"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c\") " pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:08:23.167438 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:23.167358 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6vrc\"" Sep 18 21:08:23.175765 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:23.175743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt75r" Sep 18 21:08:23.449874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:23.449678 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rt75r"] Sep 18 21:08:23.452393 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:08:23.452366 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e35eaad_1b7d_40d2_a280_ef6d3c538a8c.slice/crio-c831ffe840e00884f53966c3d28b3962f6e44745d889361541eb216f8f041a92 WatchSource:0}: Error finding container c831ffe840e00884f53966c3d28b3962f6e44745d889361541eb216f8f041a92: Status 404 returned error can't find the container with id c831ffe840e00884f53966c3d28b3962f6e44745d889361541eb216f8f041a92 Sep 18 21:08:23.894460 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:23.894427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rt75r" event={"ID":"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c","Type":"ContainerStarted","Data":"c831ffe840e00884f53966c3d28b3962f6e44745d889361541eb216f8f041a92"} Sep 18 21:08:24.900013 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:24.899984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rt75r" event={"ID":"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c","Type":"ContainerStarted","Data":"beabd3b829bfa13215657c904730095b151d1c9a961a23a2faa195bf3dfe56f9"} Sep 18 21:08:25.658460 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:25.658424 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:25.906274 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:25.906226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rt75r" event={"ID":"9e35eaad-1b7d-40d2-a280-ef6d3c538a8c","Type":"ContainerStarted","Data":"7870756f7bb5f69123843f4320efbbc8f29c9dec60a22aa4d2782a51c0a651e3"} Sep 18 21:08:25.924822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:25.924709 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rt75r" podStartSLOduration=131.688294218 podStartE2EDuration="2m12.924689338s" totalImagesPullingTime="1.23639512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:25.923356876 +0000 UTC m=+134.853102752" watchObservedRunningTime="2026-09-18 21:08:25.924689338 +0000 UTC m=+134.854435211" Sep 18 21:08:31.242044 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:31.242000 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:31.242649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:31.242072 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:31.247284 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:31.247255 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:31.934728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:31.934698 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:08:31.986937 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:31.986901 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-964796757-hlccj"] Sep 18 21:08:41.967672 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:41.967637 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ed5b13a-605a-46fe-a982-2d16520f93ea" containerID="4ffe9aa7c757268d22ba4d2cbba03f609b2aa637e49476b9f34cea81538d6103" exitCode=0 Sep 18 21:08:41.968068 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:41.967684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" event={"ID":"2ed5b13a-605a-46fe-a982-2d16520f93ea","Type":"ContainerDied","Data":"4ffe9aa7c757268d22ba4d2cbba03f609b2aa637e49476b9f34cea81538d6103"} Sep 18 21:08:41.968068 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:41.968038 2577 scope.go:117] "RemoveContainer" containerID="4ffe9aa7c757268d22ba4d2cbba03f609b2aa637e49476b9f34cea81538d6103" Sep 18 21:08:42.973102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:42.973066 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-m8g7n" event={"ID":"2ed5b13a-605a-46fe-a982-2d16520f93ea","Type":"ContainerStarted","Data":"81a0689a85c1e96b7b296983f79da236e6ad27556f5c4fcaa48f388f674acb6d"} Sep 18 21:08:45.894431 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:45.894367 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d68c4cd5d-th5b8" podUID="d67ca8e9-bd48-43ed-bb34-c25059cfc36d" containerName="console" containerID="cri-o://7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089" gracePeriod=15 Sep 18 21:08:46.137462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.137436 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:46.252011 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.251930 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6spr\" (UniqueName: \"kubernetes.io/projected/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-kube-api-access-v6spr\") pod \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " Sep 18 21:08:46.252011 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.251996 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-serving-cert\") pod \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " Sep 18 21:08:46.252209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.252022 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-service-ca\") pod \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " Sep 18 21:08:46.252209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.252073 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-config\") pod \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " Sep 18 21:08:46.252209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.252094 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-oauth-serving-cert\") pod \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " Sep 18 21:08:46.252209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.252128 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-oauth-config\") pod \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " Sep 18 21:08:46.252209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.252151 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-trusted-ca-bundle\") pod \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\" (UID: \"d67ca8e9-bd48-43ed-bb34-c25059cfc36d\") " Sep 18 21:08:46.252844 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.252791 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-config" (OuterVolumeSpecName: "console-config") pod "d67ca8e9-bd48-43ed-bb34-c25059cfc36d" (UID: "d67ca8e9-bd48-43ed-bb34-c25059cfc36d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:46.252844 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.252799 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-service-ca" (OuterVolumeSpecName: "service-ca") pod "d67ca8e9-bd48-43ed-bb34-c25059cfc36d" (UID: "d67ca8e9-bd48-43ed-bb34-c25059cfc36d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:46.253074 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.252867 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d67ca8e9-bd48-43ed-bb34-c25059cfc36d" (UID: "d67ca8e9-bd48-43ed-bb34-c25059cfc36d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:46.254800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.253303 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d67ca8e9-bd48-43ed-bb34-c25059cfc36d" (UID: "d67ca8e9-bd48-43ed-bb34-c25059cfc36d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:46.256671 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.255281 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d67ca8e9-bd48-43ed-bb34-c25059cfc36d" (UID: "d67ca8e9-bd48-43ed-bb34-c25059cfc36d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:08:46.259121 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.259090 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d67ca8e9-bd48-43ed-bb34-c25059cfc36d" (UID: "d67ca8e9-bd48-43ed-bb34-c25059cfc36d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:08:46.259121 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.259105 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-kube-api-access-v6spr" (OuterVolumeSpecName: "kube-api-access-v6spr") pod "d67ca8e9-bd48-43ed-bb34-c25059cfc36d" (UID: "d67ca8e9-bd48-43ed-bb34-c25059cfc36d"). InnerVolumeSpecName "kube-api-access-v6spr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:08:46.353590 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.353555 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-config\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:46.353590 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.353583 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-oauth-serving-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:46.353590 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.353595 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-oauth-config\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:46.353837 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.353604 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-trusted-ca-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:46.353837 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.353614 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6spr\" (UniqueName: \"kubernetes.io/projected/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-kube-api-access-v6spr\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:46.353837 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.353623 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-console-serving-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:46.353837 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.353631 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67ca8e9-bd48-43ed-bb34-c25059cfc36d-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:46.990170 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.990131 2577 generic.go:356] "Generic (PLEG): container finished" podID="d67ca8e9-bd48-43ed-bb34-c25059cfc36d" containerID="7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089" exitCode=0 Sep 18 21:08:46.990631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.990195 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d68c4cd5d-th5b8" Sep 18 21:08:46.990631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.990220 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d68c4cd5d-th5b8" event={"ID":"d67ca8e9-bd48-43ed-bb34-c25059cfc36d","Type":"ContainerDied","Data":"7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089"} Sep 18 21:08:46.990631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.990290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d68c4cd5d-th5b8" event={"ID":"d67ca8e9-bd48-43ed-bb34-c25059cfc36d","Type":"ContainerDied","Data":"d4443aa4c5c25b3184dc8350c27580795b41c821575a34b8408a35429f225038"} Sep 18 21:08:46.990631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.990321 2577 scope.go:117] "RemoveContainer" containerID="7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089" Sep 18 21:08:46.999113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.999096 2577 scope.go:117] "RemoveContainer" containerID="7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089" Sep 18 21:08:46.999374 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:08:46.999355 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089\": container with ID starting with 7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089 not found: ID does not exist" containerID="7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089" Sep 18 21:08:46.999436 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:46.999385 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089"} err="failed to get container status \"7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089\": rpc error: code = NotFound desc = could not find container \"7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089\": container with ID starting with 7cf5387095f2d6280d41bb19fece9c36511e29688f04e33ca331da836ea51089 not found: ID does not exist" Sep 18 21:08:47.013995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:47.013972 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d68c4cd5d-th5b8"] Sep 18 21:08:47.019688 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:47.019666 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d68c4cd5d-th5b8"] Sep 18 21:08:47.357206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:47.357174 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d67ca8e9-bd48-43ed-bb34-c25059cfc36d" path="/var/lib/kubelet/pods/d67ca8e9-bd48-43ed-bb34-c25059cfc36d/volumes" Sep 18 21:08:57.014933 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.014847 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-964796757-hlccj" podUID="bd57065c-a1b2-4e3f-aba6-d897f8dac6db" containerName="console" containerID="cri-o://751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3" gracePeriod=15 Sep 18 21:08:57.266065 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.266011 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-964796757-hlccj" Sep 18 21:08:57.347214 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347183 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-kube-api-access-n9m5k\") pod \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " Sep 18 21:08:57.347373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347227 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-oauth-serving-cert\") pod \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " Sep 18 21:08:57.347373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347254 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-oauth-config\") pod \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " Sep 18 21:08:57.347373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347313 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-trusted-ca-bundle\") pod \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " Sep 18 21:08:57.347373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347348 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-config\") pod \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " Sep 18 21:08:57.347557 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347383 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-serving-cert\") pod \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " Sep 18 21:08:57.347557 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347428 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-service-ca\") pod \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\" (UID: \"bd57065c-a1b2-4e3f-aba6-d897f8dac6db\") " Sep 18 21:08:57.347703 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347654 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bd57065c-a1b2-4e3f-aba6-d897f8dac6db" (UID: "bd57065c-a1b2-4e3f-aba6-d897f8dac6db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:57.347873 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347846 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-config" (OuterVolumeSpecName: "console-config") pod "bd57065c-a1b2-4e3f-aba6-d897f8dac6db" (UID: "bd57065c-a1b2-4e3f-aba6-d897f8dac6db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:57.347943 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347906 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-service-ca" (OuterVolumeSpecName: "service-ca") pod "bd57065c-a1b2-4e3f-aba6-d897f8dac6db" (UID: "bd57065c-a1b2-4e3f-aba6-d897f8dac6db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:57.347943 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.347912 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bd57065c-a1b2-4e3f-aba6-d897f8dac6db" (UID: "bd57065c-a1b2-4e3f-aba6-d897f8dac6db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:08:57.349476 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.349449 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bd57065c-a1b2-4e3f-aba6-d897f8dac6db" (UID: "bd57065c-a1b2-4e3f-aba6-d897f8dac6db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:08:57.349879 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.349854 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bd57065c-a1b2-4e3f-aba6-d897f8dac6db" (UID: "bd57065c-a1b2-4e3f-aba6-d897f8dac6db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:08:57.349964 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.349874 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-kube-api-access-n9m5k" (OuterVolumeSpecName: "kube-api-access-n9m5k") pod "bd57065c-a1b2-4e3f-aba6-d897f8dac6db" (UID: "bd57065c-a1b2-4e3f-aba6-d897f8dac6db"). InnerVolumeSpecName "kube-api-access-n9m5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:08:57.448292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.448253 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-oauth-serving-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:57.448292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.448285 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-oauth-config\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:57.448292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.448295 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-trusted-ca-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:57.448292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.448304 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-config\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:57.448540 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.448313 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-console-serving-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:57.448540 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.448345 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:57.448540 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:57.448355 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9m5k\" (UniqueName: \"kubernetes.io/projected/bd57065c-a1b2-4e3f-aba6-d897f8dac6db-kube-api-access-n9m5k\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:08:58.042613 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.042580 2577 generic.go:356] "Generic (PLEG): container finished" podID="bd57065c-a1b2-4e3f-aba6-d897f8dac6db" containerID="751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3" exitCode=0 Sep 18 21:08:58.043042 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.042627 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-964796757-hlccj" Sep 18 21:08:58.043042 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.042662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-964796757-hlccj" event={"ID":"bd57065c-a1b2-4e3f-aba6-d897f8dac6db","Type":"ContainerDied","Data":"751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3"} Sep 18 21:08:58.043042 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.042712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-964796757-hlccj" event={"ID":"bd57065c-a1b2-4e3f-aba6-d897f8dac6db","Type":"ContainerDied","Data":"fcf148fa210ea1082c535c940d909ef542b3d23fd19f60ba8ba942dc9cc13761"} Sep 18 21:08:58.043042 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.042731 2577 scope.go:117] "RemoveContainer" containerID="751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3" Sep 18 21:08:58.050575 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.050560 2577 scope.go:117] "RemoveContainer" containerID="751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3" Sep 18 21:08:58.050846 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:08:58.050828 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3\": container with ID starting with 751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3 not found: ID does not exist" containerID="751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3" Sep 18 21:08:58.050917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.050855 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3"} err="failed to get container status \"751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3\": rpc error: code = NotFound desc = could not find container \"751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3\": container with ID starting with 751cd09b27dee69680fe6d4f233f121b08a1a77a13a65814f9bfafa4cc4544b3 not found: ID does not exist" Sep 18 21:08:58.059318 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.059296 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-964796757-hlccj"] Sep 18 21:08:58.064714 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:58.064694 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-964796757-hlccj"] Sep 18 21:08:59.356874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:08:59.356840 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd57065c-a1b2-4e3f-aba6-d897f8dac6db" path="/var/lib/kubelet/pods/bd57065c-a1b2-4e3f-aba6-d897f8dac6db/volumes" Sep 18 21:09:38.908414 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.908383 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f6f8797c8-w8wmq"] Sep 18 21:09:38.908886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.908698 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd57065c-a1b2-4e3f-aba6-d897f8dac6db" containerName="console" Sep 18 21:09:38.908886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.908710 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd57065c-a1b2-4e3f-aba6-d897f8dac6db" containerName="console" Sep 18 21:09:38.908886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.908729 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67ca8e9-bd48-43ed-bb34-c25059cfc36d" containerName="console" Sep 18 21:09:38.908886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.908734 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67ca8e9-bd48-43ed-bb34-c25059cfc36d" containerName="console" Sep 18 21:09:38.908886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.908785 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67ca8e9-bd48-43ed-bb34-c25059cfc36d" containerName="console" Sep 18 21:09:38.908886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.908793 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd57065c-a1b2-4e3f-aba6-d897f8dac6db" containerName="console" Sep 18 21:09:38.910461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.910445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:38.922388 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:38.922367 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f6f8797c8-w8wmq"] Sep 18 21:09:39.094312 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.094266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-oauth-serving-cert\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.094495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.094325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-config\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.094495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.094398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-serving-cert\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.094495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.094433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5pj\" (UniqueName: \"kubernetes.io/projected/566ad78a-4eb0-4ac0-99ad-c356141898aa-kube-api-access-fp5pj\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.094495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.094452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-trusted-ca-bundle\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.094495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.094493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-oauth-config\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.094663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.094590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-service-ca\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.195509 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.195417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-oauth-serving-cert\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.195509 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.195466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-config\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.195696 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.195588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-serving-cert\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.195696 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.195628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5pj\" (UniqueName: \"kubernetes.io/projected/566ad78a-4eb0-4ac0-99ad-c356141898aa-kube-api-access-fp5pj\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.195696 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.195655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-trusted-ca-bundle\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.195696 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.195691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-oauth-config\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.195898 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.195779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-service-ca\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.196336 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.196304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-oauth-serving-cert\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.196336 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.196328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-config\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.196512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.196416 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-service-ca\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.196611 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.196584 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-trusted-ca-bundle\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.198198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.198168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-oauth-config\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.198311 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.198228 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-serving-cert\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.204299 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.204270 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5pj\" (UniqueName: \"kubernetes.io/projected/566ad78a-4eb0-4ac0-99ad-c356141898aa-kube-api-access-fp5pj\") pod \"console-6f6f8797c8-w8wmq\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.219127 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.219084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:39.491326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:39.491244 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f6f8797c8-w8wmq"] Sep 18 21:09:39.494178 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:09:39.494151 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566ad78a_4eb0_4ac0_99ad_c356141898aa.slice/crio-a85f8b40b8a799e813072845924568307fc06b76f99f40d1ccbb37078e9beed8 WatchSource:0}: Error finding container a85f8b40b8a799e813072845924568307fc06b76f99f40d1ccbb37078e9beed8: Status 404 returned error can't find the container with id a85f8b40b8a799e813072845924568307fc06b76f99f40d1ccbb37078e9beed8 Sep 18 21:09:40.210792 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:40.210759 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6f8797c8-w8wmq" event={"ID":"566ad78a-4eb0-4ac0-99ad-c356141898aa","Type":"ContainerStarted","Data":"5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50"} Sep 18 21:09:40.211161 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:40.210811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6f8797c8-w8wmq" event={"ID":"566ad78a-4eb0-4ac0-99ad-c356141898aa","Type":"ContainerStarted","Data":"a85f8b40b8a799e813072845924568307fc06b76f99f40d1ccbb37078e9beed8"} Sep 18 21:09:40.241006 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:40.240922 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f6f8797c8-w8wmq" podStartSLOduration=2.240906153 podStartE2EDuration="2.240906153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:09:40.240546529 +0000 UTC m=+209.170292404" watchObservedRunningTime="2026-09-18 21:09:40.240906153 +0000 UTC m=+209.170652026" Sep 18 21:09:49.219882 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:49.219836 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:49.220312 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:49.219922 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:49.224708 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:49.224684 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:49.246111 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:49.246089 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:09:49.294409 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:09:49.294381 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc6795779-cszfk"] Sep 18 21:10:14.312775 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.312724 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bc6795779-cszfk" podUID="7b8e79c8-ff79-4deb-a577-0a273da5bb13" containerName="console" containerID="cri-o://d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562" gracePeriod=15 Sep 18 21:10:14.555881 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.555856 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:10:14.682175 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682078 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpzrr\" (UniqueName: \"kubernetes.io/projected/7b8e79c8-ff79-4deb-a577-0a273da5bb13-kube-api-access-gpzrr\") pod \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " Sep 18 21:10:14.682175 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682128 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-service-ca\") pod \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " Sep 18 21:10:14.682175 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682167 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-oauth-serving-cert\") pod \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " Sep 18 21:10:14.682447 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682192 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-config\") pod \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " Sep 18 21:10:14.682447 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682234 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-serving-cert\") pod \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " Sep 18 21:10:14.682447 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682405 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-trusted-ca-bundle\") pod \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " Sep 18 21:10:14.682601 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682478 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-oauth-config\") pod \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\" (UID: \"7b8e79c8-ff79-4deb-a577-0a273da5bb13\") " Sep 18 21:10:14.682601 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682580 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-service-ca" (OuterVolumeSpecName: "service-ca") pod "7b8e79c8-ff79-4deb-a577-0a273da5bb13" (UID: "7b8e79c8-ff79-4deb-a577-0a273da5bb13"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:10:14.682696 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682673 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-config" (OuterVolumeSpecName: "console-config") pod "7b8e79c8-ff79-4deb-a577-0a273da5bb13" (UID: "7b8e79c8-ff79-4deb-a577-0a273da5bb13"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:10:14.682747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682727 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7b8e79c8-ff79-4deb-a577-0a273da5bb13" (UID: "7b8e79c8-ff79-4deb-a577-0a273da5bb13"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:10:14.682860 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682817 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7b8e79c8-ff79-4deb-a577-0a273da5bb13" (UID: "7b8e79c8-ff79-4deb-a577-0a273da5bb13"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:10:14.682860 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682841 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:14.682860 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682858 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-oauth-serving-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:14.683073 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.682870 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-config\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:14.684475 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.684453 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7b8e79c8-ff79-4deb-a577-0a273da5bb13" (UID: "7b8e79c8-ff79-4deb-a577-0a273da5bb13"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:10:14.684821 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.684795 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7b8e79c8-ff79-4deb-a577-0a273da5bb13" (UID: "7b8e79c8-ff79-4deb-a577-0a273da5bb13"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:10:14.684821 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.684803 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8e79c8-ff79-4deb-a577-0a273da5bb13-kube-api-access-gpzrr" (OuterVolumeSpecName: "kube-api-access-gpzrr") pod "7b8e79c8-ff79-4deb-a577-0a273da5bb13" (UID: "7b8e79c8-ff79-4deb-a577-0a273da5bb13"). InnerVolumeSpecName "kube-api-access-gpzrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:10:14.783515 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.783477 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-oauth-config\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:14.783515 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.783507 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpzrr\" (UniqueName: \"kubernetes.io/projected/7b8e79c8-ff79-4deb-a577-0a273da5bb13-kube-api-access-gpzrr\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:14.783515 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.783517 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b8e79c8-ff79-4deb-a577-0a273da5bb13-console-serving-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:14.783515 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:14.783525 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8e79c8-ff79-4deb-a577-0a273da5bb13-trusted-ca-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:15.340439 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.340406 2577 generic.go:356] "Generic (PLEG): container finished" podID="7b8e79c8-ff79-4deb-a577-0a273da5bb13" containerID="d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562" exitCode=0 Sep 18 21:10:15.340855 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.340455 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc6795779-cszfk" Sep 18 21:10:15.340855 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.340485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc6795779-cszfk" event={"ID":"7b8e79c8-ff79-4deb-a577-0a273da5bb13","Type":"ContainerDied","Data":"d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562"} Sep 18 21:10:15.340855 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.340535 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc6795779-cszfk" event={"ID":"7b8e79c8-ff79-4deb-a577-0a273da5bb13","Type":"ContainerDied","Data":"e64b9cd2b7f868f37d0621dcb824e64a60c781aafc8946ac507fb777c544251e"} Sep 18 21:10:15.340855 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.340557 2577 scope.go:117] "RemoveContainer" containerID="d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562" Sep 18 21:10:15.348995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.348980 2577 scope.go:117] "RemoveContainer" containerID="d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562" Sep 18 21:10:15.349285 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:10:15.349266 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562\": container with ID starting with d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562 not found: ID does not exist" containerID="d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562" Sep 18 21:10:15.349373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.349296 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562"} err="failed to get container status \"d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562\": rpc error: code = NotFound desc = could not find container \"d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562\": container with ID starting with d69d42c6021c3a4575c7a0d7dac72a15cc676f91c27b71be07e927c58fe8a562 not found: ID does not exist" Sep 18 21:10:15.361396 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.361368 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc6795779-cszfk"] Sep 18 21:10:15.365132 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:15.365106 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bc6795779-cszfk"] Sep 18 21:10:17.357205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:17.357172 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b8e79c8-ff79-4deb-a577-0a273da5bb13" path="/var/lib/kubelet/pods/7b8e79c8-ff79-4deb-a577-0a273da5bb13/volumes" Sep 18 21:10:39.154920 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.154828 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b"] Sep 18 21:10:39.155402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.155292 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b8e79c8-ff79-4deb-a577-0a273da5bb13" containerName="console" Sep 18 21:10:39.155402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.155308 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b8e79c8-ff79-4deb-a577-0a273da5bb13" containerName="console" Sep 18 21:10:39.155402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.155382 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b8e79c8-ff79-4deb-a577-0a273da5bb13" containerName="console" Sep 18 21:10:39.157603 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.157583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.163169 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.163149 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 21:10:39.163301 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.163188 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvq7g\"" Sep 18 21:10:39.164040 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.163820 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 21:10:39.168281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.168258 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b"] Sep 18 21:10:39.189776 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.189750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.189885 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.189780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.189885 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.189828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbmn\" (UniqueName: \"kubernetes.io/projected/694cb4fb-c949-41c8-8109-1fb03f4046b2-kube-api-access-lfbmn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.290942 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.290908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.290942 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.290942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.291166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.290997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbmn\" (UniqueName: \"kubernetes.io/projected/694cb4fb-c949-41c8-8109-1fb03f4046b2-kube-api-access-lfbmn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.291282 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.291264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.291331 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.291294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.300383 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.300352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbmn\" (UniqueName: \"kubernetes.io/projected/694cb4fb-c949-41c8-8109-1fb03f4046b2-kube-api-access-lfbmn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.466709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.466623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:39.737139 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:39.737110 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b"] Sep 18 21:10:39.739543 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:10:39.739517 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694cb4fb_c949_41c8_8109_1fb03f4046b2.slice/crio-31213c8021eb9f53d188abb089ff1de0a8fd0cb584808b70a0bb6c8447b7b1f8 WatchSource:0}: Error finding container 31213c8021eb9f53d188abb089ff1de0a8fd0cb584808b70a0bb6c8447b7b1f8: Status 404 returned error can't find the container with id 31213c8021eb9f53d188abb089ff1de0a8fd0cb584808b70a0bb6c8447b7b1f8 Sep 18 21:10:40.437834 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:40.437797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" event={"ID":"694cb4fb-c949-41c8-8109-1fb03f4046b2","Type":"ContainerStarted","Data":"31213c8021eb9f53d188abb089ff1de0a8fd0cb584808b70a0bb6c8447b7b1f8"} Sep 18 21:10:46.468917 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:46.468879 2577 generic.go:356] "Generic (PLEG): container finished" podID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerID="39c323a835bf0fdad24e8ea97c51b0a119541b18eeaae05a90a5db58596d8c39" exitCode=0 Sep 18 21:10:46.469459 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:46.468983 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" event={"ID":"694cb4fb-c949-41c8-8109-1fb03f4046b2","Type":"ContainerDied","Data":"39c323a835bf0fdad24e8ea97c51b0a119541b18eeaae05a90a5db58596d8c39"} Sep 18 21:10:48.485653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:48.485622 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" event={"ID":"694cb4fb-c949-41c8-8109-1fb03f4046b2","Type":"ContainerStarted","Data":"a7de1003bd3a91b5ea7e311ca723316103610823b81ac88fd3ac9a19e6936480"} Sep 18 21:10:49.491126 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:49.491088 2577 generic.go:356] "Generic (PLEG): container finished" podID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerID="a7de1003bd3a91b5ea7e311ca723316103610823b81ac88fd3ac9a19e6936480" exitCode=0 Sep 18 21:10:49.491595 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:49.491157 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" event={"ID":"694cb4fb-c949-41c8-8109-1fb03f4046b2","Type":"ContainerDied","Data":"a7de1003bd3a91b5ea7e311ca723316103610823b81ac88fd3ac9a19e6936480"} Sep 18 21:10:57.524204 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:57.524166 2577 generic.go:356] "Generic (PLEG): container finished" podID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerID="e12781a5a9f7cd4b8549217e3380a0ed07e88602db5b9130b82891846a5b9bad" exitCode=0 Sep 18 21:10:57.524606 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:57.524239 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" event={"ID":"694cb4fb-c949-41c8-8109-1fb03f4046b2","Type":"ContainerDied","Data":"e12781a5a9f7cd4b8549217e3380a0ed07e88602db5b9130b82891846a5b9bad"} Sep 18 21:10:58.645574 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.645550 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:58.763586 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.763555 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-util\") pod \"694cb4fb-c949-41c8-8109-1fb03f4046b2\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " Sep 18 21:10:58.763752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.763613 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-bundle\") pod \"694cb4fb-c949-41c8-8109-1fb03f4046b2\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " Sep 18 21:10:58.763752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.763662 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfbmn\" (UniqueName: \"kubernetes.io/projected/694cb4fb-c949-41c8-8109-1fb03f4046b2-kube-api-access-lfbmn\") pod \"694cb4fb-c949-41c8-8109-1fb03f4046b2\" (UID: \"694cb4fb-c949-41c8-8109-1fb03f4046b2\") " Sep 18 21:10:58.764385 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.764357 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-bundle" (OuterVolumeSpecName: "bundle") pod "694cb4fb-c949-41c8-8109-1fb03f4046b2" (UID: "694cb4fb-c949-41c8-8109-1fb03f4046b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:10:58.765916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.765887 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694cb4fb-c949-41c8-8109-1fb03f4046b2-kube-api-access-lfbmn" (OuterVolumeSpecName: "kube-api-access-lfbmn") pod "694cb4fb-c949-41c8-8109-1fb03f4046b2" (UID: "694cb4fb-c949-41c8-8109-1fb03f4046b2"). InnerVolumeSpecName "kube-api-access-lfbmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:10:58.767725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.767704 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-util" (OuterVolumeSpecName: "util") pod "694cb4fb-c949-41c8-8109-1fb03f4046b2" (UID: "694cb4fb-c949-41c8-8109-1fb03f4046b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:10:58.865047 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.865025 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:58.865047 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.865046 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfbmn\" (UniqueName: \"kubernetes.io/projected/694cb4fb-c949-41c8-8109-1fb03f4046b2-kube-api-access-lfbmn\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:58.865192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:58.865057 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/694cb4fb-c949-41c8-8109-1fb03f4046b2-util\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:10:59.534095 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:59.534066 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" Sep 18 21:10:59.534252 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:59.534091 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt56b" event={"ID":"694cb4fb-c949-41c8-8109-1fb03f4046b2","Type":"ContainerDied","Data":"31213c8021eb9f53d188abb089ff1de0a8fd0cb584808b70a0bb6c8447b7b1f8"} Sep 18 21:10:59.534252 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:10:59.534143 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31213c8021eb9f53d188abb089ff1de0a8fd0cb584808b70a0bb6c8447b7b1f8" Sep 18 21:11:08.041592 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.041552 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs"] Sep 18 21:11:08.042077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.041845 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerName="extract" Sep 18 21:11:08.042077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.041857 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerName="extract" Sep 18 21:11:08.042077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.041869 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerName="pull" Sep 18 21:11:08.042077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.041875 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerName="pull" Sep 18 21:11:08.042077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.041895 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerName="util" Sep 18 21:11:08.042077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.041901 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerName="util" Sep 18 21:11:08.042077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.041944 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="694cb4fb-c949-41c8-8109-1fb03f4046b2" containerName="extract" Sep 18 21:11:08.045040 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.045021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" Sep 18 21:11:08.047641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.047619 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6mhph\"" Sep 18 21:11:08.047641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.047621 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 18 21:11:08.048497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.048482 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 18 21:11:08.072051 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.072026 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs"] Sep 18 21:11:08.139976 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.139925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slf92\" (UniqueName: \"kubernetes.io/projected/df6942cd-4631-407c-a66d-90b0790a507e-kube-api-access-slf92\") pod \"cert-manager-operator-controller-manager-55c444bf86-8s5fs\" (UID: \"df6942cd-4631-407c-a66d-90b0790a507e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" Sep 18 21:11:08.139976 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.139980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df6942cd-4631-407c-a66d-90b0790a507e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8s5fs\" (UID: \"df6942cd-4631-407c-a66d-90b0790a507e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" Sep 18 21:11:08.240586 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.240556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf92\" (UniqueName: \"kubernetes.io/projected/df6942cd-4631-407c-a66d-90b0790a507e-kube-api-access-slf92\") pod \"cert-manager-operator-controller-manager-55c444bf86-8s5fs\" (UID: \"df6942cd-4631-407c-a66d-90b0790a507e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" Sep 18 21:11:08.240586 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.240587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df6942cd-4631-407c-a66d-90b0790a507e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8s5fs\" (UID: \"df6942cd-4631-407c-a66d-90b0790a507e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" Sep 18 21:11:08.240927 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.240913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df6942cd-4631-407c-a66d-90b0790a507e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8s5fs\" (UID: \"df6942cd-4631-407c-a66d-90b0790a507e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" Sep 18 21:11:08.251522 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.251496 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slf92\" (UniqueName: \"kubernetes.io/projected/df6942cd-4631-407c-a66d-90b0790a507e-kube-api-access-slf92\") pod \"cert-manager-operator-controller-manager-55c444bf86-8s5fs\" (UID: \"df6942cd-4631-407c-a66d-90b0790a507e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" Sep 18 21:11:08.353434 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.353404 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" Sep 18 21:11:08.651552 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:08.651524 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs"] Sep 18 21:11:08.654183 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:11:08.654157 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6942cd_4631_407c_a66d_90b0790a507e.slice/crio-6dc6f5f33b54aebd12a98890ada11a4873ed123ca31ede13e0687b5581b73748 WatchSource:0}: Error finding container 6dc6f5f33b54aebd12a98890ada11a4873ed123ca31ede13e0687b5581b73748: Status 404 returned error can't find the container with id 6dc6f5f33b54aebd12a98890ada11a4873ed123ca31ede13e0687b5581b73748 Sep 18 21:11:09.574479 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:09.574435 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" event={"ID":"df6942cd-4631-407c-a66d-90b0790a507e","Type":"ContainerStarted","Data":"6dc6f5f33b54aebd12a98890ada11a4873ed123ca31ede13e0687b5581b73748"} Sep 18 21:11:10.580560 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:10.580521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" event={"ID":"df6942cd-4631-407c-a66d-90b0790a507e","Type":"ContainerStarted","Data":"58c6869ed97f958b9d270dd9eeb6b2182b69747b95b4aeddb042c5c5b04f5237"} Sep 18 21:11:10.606162 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:10.606106 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8s5fs" podStartSLOduration=0.854861061 podStartE2EDuration="2.606087837s" totalImagesPullingTime="1.751226776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:10.604154525 +0000 UTC m=+299.533900399" watchObservedRunningTime="2026-09-18 21:11:10.606087837 +0000 UTC m=+299.535833714" Sep 18 21:11:11.461440 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.461401 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg"] Sep 18 21:11:11.463898 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.463879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.477368 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.477341 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvq7g\"" Sep 18 21:11:11.496563 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.496523 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 21:11:11.496741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.496523 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 21:11:11.498800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.498769 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg"] Sep 18 21:11:11.570081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.570043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.570276 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.570112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.570276 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.570176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdv6p\" (UniqueName: \"kubernetes.io/projected/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-kube-api-access-hdv6p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.670774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.670736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.671287 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.670829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.671287 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.670864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdv6p\" (UniqueName: \"kubernetes.io/projected/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-kube-api-access-hdv6p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.671287 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.671176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.671287 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.671180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.707440 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.707406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdv6p\" (UniqueName: \"kubernetes.io/projected/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-kube-api-access-hdv6p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.772887 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.772799 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:11.940833 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.940797 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kqhjz"] Sep 18 21:11:11.943107 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.943090 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:11.947611 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.947590 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 18 21:11:11.947734 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.947624 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 18 21:11:11.948572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.948554 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-psxbn\"" Sep 18 21:11:11.956069 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:11.956044 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kqhjz"] Sep 18 21:11:12.047405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.047381 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg"] Sep 18 21:11:12.049879 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:11:12.049855 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce919ee5_a80e_4225_bf63_9d29bf9d0be6.slice/crio-bd0c439ba8425b2782b0b168b1d7cd751beecd4b641bbc3a4d79c288840a288e WatchSource:0}: Error finding container bd0c439ba8425b2782b0b168b1d7cd751beecd4b641bbc3a4d79c288840a288e: Status 404 returned error can't find the container with id bd0c439ba8425b2782b0b168b1d7cd751beecd4b641bbc3a4d79c288840a288e Sep 18 21:11:12.074095 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.074068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmb8\" (UniqueName: \"kubernetes.io/projected/810934be-076c-4d8f-9b3b-7f28fadc368c-kube-api-access-9pmb8\") pod \"cert-manager-webhook-6486ffd997-kqhjz\" (UID: \"810934be-076c-4d8f-9b3b-7f28fadc368c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:12.074199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.074138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/810934be-076c-4d8f-9b3b-7f28fadc368c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kqhjz\" (UID: \"810934be-076c-4d8f-9b3b-7f28fadc368c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:12.174919 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.174894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/810934be-076c-4d8f-9b3b-7f28fadc368c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kqhjz\" (UID: \"810934be-076c-4d8f-9b3b-7f28fadc368c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:12.175068 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.174981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmb8\" (UniqueName: \"kubernetes.io/projected/810934be-076c-4d8f-9b3b-7f28fadc368c-kube-api-access-9pmb8\") pod \"cert-manager-webhook-6486ffd997-kqhjz\" (UID: \"810934be-076c-4d8f-9b3b-7f28fadc368c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:12.183685 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.183661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmb8\" (UniqueName: \"kubernetes.io/projected/810934be-076c-4d8f-9b3b-7f28fadc368c-kube-api-access-9pmb8\") pod \"cert-manager-webhook-6486ffd997-kqhjz\" (UID: \"810934be-076c-4d8f-9b3b-7f28fadc368c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:12.184854 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.184836 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/810934be-076c-4d8f-9b3b-7f28fadc368c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kqhjz\" (UID: \"810934be-076c-4d8f-9b3b-7f28fadc368c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:12.259653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.259617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:12.548974 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.548927 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kqhjz"] Sep 18 21:11:12.550912 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:11:12.550886 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod810934be_076c_4d8f_9b3b_7f28fadc368c.slice/crio-d4bd488637a50743dfafa87370d580fdd6cfd767f29bdbf75b919188f479b05e WatchSource:0}: Error finding container d4bd488637a50743dfafa87370d580fdd6cfd767f29bdbf75b919188f479b05e: Status 404 returned error can't find the container with id d4bd488637a50743dfafa87370d580fdd6cfd767f29bdbf75b919188f479b05e Sep 18 21:11:12.590337 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.590304 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerID="e94ebd490153b71d8196b5e0ed4581520c582fbdf311882102bf6af80430b1bc" exitCode=0 Sep 18 21:11:12.590471 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.590380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" event={"ID":"ce919ee5-a80e-4225-bf63-9d29bf9d0be6","Type":"ContainerDied","Data":"e94ebd490153b71d8196b5e0ed4581520c582fbdf311882102bf6af80430b1bc"} Sep 18 21:11:12.590471 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.590429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" event={"ID":"ce919ee5-a80e-4225-bf63-9d29bf9d0be6","Type":"ContainerStarted","Data":"bd0c439ba8425b2782b0b168b1d7cd751beecd4b641bbc3a4d79c288840a288e"} Sep 18 21:11:12.591486 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:12.591466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" event={"ID":"810934be-076c-4d8f-9b3b-7f28fadc368c","Type":"ContainerStarted","Data":"d4bd488637a50743dfafa87370d580fdd6cfd767f29bdbf75b919188f479b05e"} Sep 18 21:11:13.245760 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:13.245734 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:11:13.246653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:13.246606 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:11:13.257197 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:13.257177 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 21:11:15.611380 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:15.611347 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" event={"ID":"ce919ee5-a80e-4225-bf63-9d29bf9d0be6","Type":"ContainerStarted","Data":"e789b159fc5b1022add48f836fbcba72b25bd0336ad1eaa26554d08a66861e11"} Sep 18 21:11:15.613046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:15.613007 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" event={"ID":"810934be-076c-4d8f-9b3b-7f28fadc368c","Type":"ContainerStarted","Data":"dc3315fcf8cf28ea4079e9c928b82038200477887ecdbebc027216702e699f0d"} Sep 18 21:11:15.613192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:15.613143 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:15.645121 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:15.645068 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" podStartSLOduration=1.6907004460000001 podStartE2EDuration="4.645051135s" totalImagesPullingTime="2.954350689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:15.643869542 +0000 UTC m=+304.573615414" watchObservedRunningTime="2026-09-18 21:11:15.645051135 +0000 UTC m=+304.574797008" Sep 18 21:11:16.619053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:16.619016 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerID="e789b159fc5b1022add48f836fbcba72b25bd0336ad1eaa26554d08a66861e11" exitCode=0 Sep 18 21:11:16.619462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:16.619103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" event={"ID":"ce919ee5-a80e-4225-bf63-9d29bf9d0be6","Type":"ContainerDied","Data":"e789b159fc5b1022add48f836fbcba72b25bd0336ad1eaa26554d08a66861e11"} Sep 18 21:11:17.625187 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:17.625150 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerID="0d22d70011523ba368e298ae5e490c3116f022f0b4eac8c91a342086b5996730" exitCode=0 Sep 18 21:11:17.625573 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:17.625226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" event={"ID":"ce919ee5-a80e-4225-bf63-9d29bf9d0be6","Type":"ContainerDied","Data":"0d22d70011523ba368e298ae5e490c3116f022f0b4eac8c91a342086b5996730"} Sep 18 21:11:18.751908 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:18.751886 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:18.931806 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:18.931719 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdv6p\" (UniqueName: \"kubernetes.io/projected/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-kube-api-access-hdv6p\") pod \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " Sep 18 21:11:18.931806 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:18.931785 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-bundle\") pod \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " Sep 18 21:11:18.932015 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:18.931899 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-util\") pod \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\" (UID: \"ce919ee5-a80e-4225-bf63-9d29bf9d0be6\") " Sep 18 21:11:18.932200 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:18.932175 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-bundle" (OuterVolumeSpecName: "bundle") pod "ce919ee5-a80e-4225-bf63-9d29bf9d0be6" (UID: "ce919ee5-a80e-4225-bf63-9d29bf9d0be6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:11:18.933861 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:18.933829 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-kube-api-access-hdv6p" (OuterVolumeSpecName: "kube-api-access-hdv6p") pod "ce919ee5-a80e-4225-bf63-9d29bf9d0be6" (UID: "ce919ee5-a80e-4225-bf63-9d29bf9d0be6"). InnerVolumeSpecName "kube-api-access-hdv6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:11:18.937147 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:18.937127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-util" (OuterVolumeSpecName: "util") pod "ce919ee5-a80e-4225-bf63-9d29bf9d0be6" (UID: "ce919ee5-a80e-4225-bf63-9d29bf9d0be6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:11:19.032692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.032659 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-util\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:19.032692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.032688 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdv6p\" (UniqueName: \"kubernetes.io/projected/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-kube-api-access-hdv6p\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:19.032692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.032698 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce919ee5-a80e-4225-bf63-9d29bf9d0be6-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:19.111475 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.111442 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2"] Sep 18 21:11:19.111730 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.111720 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerName="extract" Sep 18 21:11:19.111777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.111733 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerName="extract" Sep 18 21:11:19.111777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.111746 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerName="pull" Sep 18 21:11:19.111777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.111752 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerName="pull" Sep 18 21:11:19.111777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.111763 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerName="util" Sep 18 21:11:19.111777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.111768 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerName="util" Sep 18 21:11:19.111922 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.111821 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce919ee5-a80e-4225-bf63-9d29bf9d0be6" containerName="extract" Sep 18 21:11:19.113622 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.113605 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" Sep 18 21:11:19.116441 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.116421 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lg2fx\"" Sep 18 21:11:19.129787 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.129764 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2"] Sep 18 21:11:19.234608 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.234530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53de7c23-26bc-456f-ba19-35577c6fad3d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lcvk2\" (UID: \"53de7c23-26bc-456f-ba19-35577c6fad3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" Sep 18 21:11:19.234608 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.234585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zgm\" (UniqueName: \"kubernetes.io/projected/53de7c23-26bc-456f-ba19-35577c6fad3d-kube-api-access-c8zgm\") pod \"cert-manager-cainjector-559cdd8c87-lcvk2\" (UID: \"53de7c23-26bc-456f-ba19-35577c6fad3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" Sep 18 21:11:19.335686 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.335649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zgm\" (UniqueName: \"kubernetes.io/projected/53de7c23-26bc-456f-ba19-35577c6fad3d-kube-api-access-c8zgm\") pod \"cert-manager-cainjector-559cdd8c87-lcvk2\" (UID: \"53de7c23-26bc-456f-ba19-35577c6fad3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" Sep 18 21:11:19.335874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.335721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53de7c23-26bc-456f-ba19-35577c6fad3d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lcvk2\" (UID: \"53de7c23-26bc-456f-ba19-35577c6fad3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" Sep 18 21:11:19.344413 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.344385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zgm\" (UniqueName: \"kubernetes.io/projected/53de7c23-26bc-456f-ba19-35577c6fad3d-kube-api-access-c8zgm\") pod \"cert-manager-cainjector-559cdd8c87-lcvk2\" (UID: \"53de7c23-26bc-456f-ba19-35577c6fad3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" Sep 18 21:11:19.344537 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.344435 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53de7c23-26bc-456f-ba19-35577c6fad3d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lcvk2\" (UID: \"53de7c23-26bc-456f-ba19-35577c6fad3d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" Sep 18 21:11:19.422059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.422024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" Sep 18 21:11:19.635393 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.635358 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" event={"ID":"ce919ee5-a80e-4225-bf63-9d29bf9d0be6","Type":"ContainerDied","Data":"bd0c439ba8425b2782b0b168b1d7cd751beecd4b641bbc3a4d79c288840a288e"} Sep 18 21:11:19.635393 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.635388 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsn4mg" Sep 18 21:11:19.635584 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.635414 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd0c439ba8425b2782b0b168b1d7cd751beecd4b641bbc3a4d79c288840a288e" Sep 18 21:11:19.698376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:19.698354 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2"] Sep 18 21:11:19.700314 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:11:19.700292 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53de7c23_26bc_456f_ba19_35577c6fad3d.slice/crio-305c4676bc285345f81bb4e8046c4384b4722c54eb0027cddf037568b3f99a97 WatchSource:0}: Error finding container 305c4676bc285345f81bb4e8046c4384b4722c54eb0027cddf037568b3f99a97: Status 404 returned error can't find the container with id 305c4676bc285345f81bb4e8046c4384b4722c54eb0027cddf037568b3f99a97 Sep 18 21:11:20.641591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:20.641553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" event={"ID":"53de7c23-26bc-456f-ba19-35577c6fad3d","Type":"ContainerStarted","Data":"a8712258c7ef76210efbeb4c5ff473b7c761d1c068a9462d0c461193faf781e0"} Sep 18 21:11:20.641591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:20.641597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" event={"ID":"53de7c23-26bc-456f-ba19-35577c6fad3d","Type":"ContainerStarted","Data":"305c4676bc285345f81bb4e8046c4384b4722c54eb0027cddf037568b3f99a97"} Sep 18 21:11:20.666520 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:20.666468 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lcvk2" podStartSLOduration=1.666453507 podStartE2EDuration="1.666453507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:20.665214318 +0000 UTC m=+309.594960190" watchObservedRunningTime="2026-09-18 21:11:20.666453507 +0000 UTC m=+309.596199379" Sep 18 21:11:21.621871 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:21.621837 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kqhjz" Sep 18 21:11:31.155092 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.155057 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn"] Sep 18 21:11:31.157596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.157574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.159935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.159915 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvq7g\"" Sep 18 21:11:31.160040 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.159915 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 21:11:31.160595 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.160580 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 21:11:31.168322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.168265 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn"] Sep 18 21:11:31.232079 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.232050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82k2r\" (UniqueName: \"kubernetes.io/projected/4efd9b54-87fd-4375-9081-24fee721c228-kube-api-access-82k2r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.232257 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.232168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.232257 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.232209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.332862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.332823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.333075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.332871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.333075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.332921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82k2r\" (UniqueName: \"kubernetes.io/projected/4efd9b54-87fd-4375-9081-24fee721c228-kube-api-access-82k2r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.333244 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.333221 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.333315 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.333250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.341853 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.341827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82k2r\" (UniqueName: \"kubernetes.io/projected/4efd9b54-87fd-4375-9081-24fee721c228-kube-api-access-82k2r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.467293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.467215 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:31.736751 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:31.736721 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn"] Sep 18 21:11:31.738213 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:11:31.738183 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efd9b54_87fd_4375_9081_24fee721c228.slice/crio-013ce49b4f82ec23b214ece4f4f604d22bc24a32aa62302b6f5cb3878f109d47 WatchSource:0}: Error finding container 013ce49b4f82ec23b214ece4f4f604d22bc24a32aa62302b6f5cb3878f109d47: Status 404 returned error can't find the container with id 013ce49b4f82ec23b214ece4f4f604d22bc24a32aa62302b6f5cb3878f109d47 Sep 18 21:11:32.696841 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:32.696748 2577 generic.go:356] "Generic (PLEG): container finished" podID="4efd9b54-87fd-4375-9081-24fee721c228" containerID="0b08b67a4d252a86b3757243492b21907f754bfab5e6a974e5bf4610aae9bdab" exitCode=0 Sep 18 21:11:32.697198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:32.696839 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" event={"ID":"4efd9b54-87fd-4375-9081-24fee721c228","Type":"ContainerDied","Data":"0b08b67a4d252a86b3757243492b21907f754bfab5e6a974e5bf4610aae9bdab"} Sep 18 21:11:32.697198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:32.696890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" event={"ID":"4efd9b54-87fd-4375-9081-24fee721c228","Type":"ContainerStarted","Data":"013ce49b4f82ec23b214ece4f4f604d22bc24a32aa62302b6f5cb3878f109d47"} Sep 18 21:11:32.697313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:32.697299 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 21:11:33.703178 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:33.703144 2577 generic.go:356] "Generic (PLEG): container finished" podID="4efd9b54-87fd-4375-9081-24fee721c228" containerID="056d8687d76ab57ffeebe46dc36580e219f50127f4fe00b36fa2ab1e657955f8" exitCode=0 Sep 18 21:11:33.703675 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:33.703235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" event={"ID":"4efd9b54-87fd-4375-9081-24fee721c228","Type":"ContainerDied","Data":"056d8687d76ab57ffeebe46dc36580e219f50127f4fe00b36fa2ab1e657955f8"} Sep 18 21:11:34.709202 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:34.709169 2577 generic.go:356] "Generic (PLEG): container finished" podID="4efd9b54-87fd-4375-9081-24fee721c228" containerID="a6fbe20662755b8e09c9cf3b0f5f09126b3635de888e2d40bf7ab6ebec787e8a" exitCode=0 Sep 18 21:11:34.709575 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:34.709248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" event={"ID":"4efd9b54-87fd-4375-9081-24fee721c228","Type":"ContainerDied","Data":"a6fbe20662755b8e09c9cf3b0f5f09126b3635de888e2d40bf7ab6ebec787e8a"} Sep 18 21:11:35.837133 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.837101 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:35.866197 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.866167 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-bundle\") pod \"4efd9b54-87fd-4375-9081-24fee721c228\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " Sep 18 21:11:35.866351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.866229 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82k2r\" (UniqueName: \"kubernetes.io/projected/4efd9b54-87fd-4375-9081-24fee721c228-kube-api-access-82k2r\") pod \"4efd9b54-87fd-4375-9081-24fee721c228\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " Sep 18 21:11:35.866351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.866321 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-util\") pod \"4efd9b54-87fd-4375-9081-24fee721c228\" (UID: \"4efd9b54-87fd-4375-9081-24fee721c228\") " Sep 18 21:11:35.866913 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.866888 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-bundle" (OuterVolumeSpecName: "bundle") pod "4efd9b54-87fd-4375-9081-24fee721c228" (UID: "4efd9b54-87fd-4375-9081-24fee721c228"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:11:35.868350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.868330 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efd9b54-87fd-4375-9081-24fee721c228-kube-api-access-82k2r" (OuterVolumeSpecName: "kube-api-access-82k2r") pod "4efd9b54-87fd-4375-9081-24fee721c228" (UID: "4efd9b54-87fd-4375-9081-24fee721c228"). InnerVolumeSpecName "kube-api-access-82k2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:11:35.875547 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.875520 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-util" (OuterVolumeSpecName: "util") pod "4efd9b54-87fd-4375-9081-24fee721c228" (UID: "4efd9b54-87fd-4375-9081-24fee721c228"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:11:35.967027 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.966921 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82k2r\" (UniqueName: \"kubernetes.io/projected/4efd9b54-87fd-4375-9081-24fee721c228-kube-api-access-82k2r\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:35.967027 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.966970 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-util\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:35.967027 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:35.966980 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4efd9b54-87fd-4375-9081-24fee721c228-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:36.720257 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:36.720224 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" event={"ID":"4efd9b54-87fd-4375-9081-24fee721c228","Type":"ContainerDied","Data":"013ce49b4f82ec23b214ece4f4f604d22bc24a32aa62302b6f5cb3878f109d47"} Sep 18 21:11:36.720257 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:36.720252 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9jvn" Sep 18 21:11:36.720471 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:36.720275 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="013ce49b4f82ec23b214ece4f4f604d22bc24a32aa62302b6f5cb3878f109d47" Sep 18 21:11:48.176789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.176756 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn"] Sep 18 21:11:48.177268 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.177064 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efd9b54-87fd-4375-9081-24fee721c228" containerName="pull" Sep 18 21:11:48.177268 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.177075 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efd9b54-87fd-4375-9081-24fee721c228" containerName="pull" Sep 18 21:11:48.177268 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.177089 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efd9b54-87fd-4375-9081-24fee721c228" containerName="util" Sep 18 21:11:48.177268 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.177094 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efd9b54-87fd-4375-9081-24fee721c228" containerName="util" Sep 18 21:11:48.177268 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.177100 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efd9b54-87fd-4375-9081-24fee721c228" containerName="extract" Sep 18 21:11:48.177268 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.177107 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efd9b54-87fd-4375-9081-24fee721c228" containerName="extract" Sep 18 21:11:48.177268 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.177164 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4efd9b54-87fd-4375-9081-24fee721c228" containerName="extract" Sep 18 21:11:48.182838 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.182822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.185455 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.185430 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 18 21:11:48.185637 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.185432 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6hf49\"" Sep 18 21:11:48.185716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.185698 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 18 21:11:48.185925 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.185901 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 18 21:11:48.186042 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.185913 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 18 21:11:48.201722 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.201699 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn"] Sep 18 21:11:48.270911 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.270874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c438df90-6953-4453-8877-bbf1117bd8f7-tmp\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.271101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.270998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c438df90-6953-4453-8877-bbf1117bd8f7-manifests\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.271101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.271043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6g8w\" (UniqueName: \"kubernetes.io/projected/c438df90-6953-4453-8877-bbf1117bd8f7-kube-api-access-j6g8w\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.271181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.271105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c438df90-6953-4453-8877-bbf1117bd8f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.271181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.271124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c438df90-6953-4453-8877-bbf1117bd8f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.297601 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.297569 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt"] Sep 18 21:11:48.301298 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.301281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.303911 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.303891 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 21:11:48.304156 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.304142 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 21:11:48.304663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.304648 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvq7g\"" Sep 18 21:11:48.312655 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.312634 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt"] Sep 18 21:11:48.371923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.371891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.372145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.371936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c438df90-6953-4453-8877-bbf1117bd8f7-tmp\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.372145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.372019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drm59\" (UniqueName: \"kubernetes.io/projected/0dc94bd1-c2a3-4659-adf9-806287affcc8-kube-api-access-drm59\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.372145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.372057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c438df90-6953-4453-8877-bbf1117bd8f7-manifests\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.372145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.372119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6g8w\" (UniqueName: \"kubernetes.io/projected/c438df90-6953-4453-8877-bbf1117bd8f7-kube-api-access-j6g8w\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.372300 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.372145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.372300 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.372198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c438df90-6953-4453-8877-bbf1117bd8f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.372300 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.372221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c438df90-6953-4453-8877-bbf1117bd8f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.372438 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.372418 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c438df90-6953-4453-8877-bbf1117bd8f7-manifests\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.374219 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.374198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c438df90-6953-4453-8877-bbf1117bd8f7-tmp\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.374528 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.374511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c438df90-6953-4453-8877-bbf1117bd8f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.374643 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.374624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c438df90-6953-4453-8877-bbf1117bd8f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.380809 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.380792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6g8w\" (UniqueName: \"kubernetes.io/projected/c438df90-6953-4453-8877-bbf1117bd8f7-kube-api-access-j6g8w\") pod \"opendatahub-operator-controller-manager-75f94bc69c-sgfsn\" (UID: \"c438df90-6953-4453-8877-bbf1117bd8f7\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.473685 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.473597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drm59\" (UniqueName: \"kubernetes.io/projected/0dc94bd1-c2a3-4659-adf9-806287affcc8-kube-api-access-drm59\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.473685 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.473640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.473882 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.473688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.474064 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.474049 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.474118 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.474076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.481410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.481389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drm59\" (UniqueName: \"kubernetes.io/projected/0dc94bd1-c2a3-4659-adf9-806287affcc8-kube-api-access-drm59\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.493247 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.493225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:48.610030 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.609994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:48.771197 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.771173 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn"] Sep 18 21:11:48.887263 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:48.887236 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt"] Sep 18 21:11:48.889417 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:11:48.889386 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc94bd1_c2a3_4659_adf9_806287affcc8.slice/crio-966bcb31df09f96569bc105d53d93c3fce22ab86dbf9963a2983537b492a9cd5 WatchSource:0}: Error finding container 966bcb31df09f96569bc105d53d93c3fce22ab86dbf9963a2983537b492a9cd5: Status 404 returned error can't find the container with id 966bcb31df09f96569bc105d53d93c3fce22ab86dbf9963a2983537b492a9cd5 Sep 18 21:11:49.787249 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:49.787212 2577 generic.go:356] "Generic (PLEG): container finished" podID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerID="4c6bb5cb9736c687d688859ecf410b21278397ce939d50b3422d6430e8663ff8" exitCode=0 Sep 18 21:11:49.787720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:49.787287 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" event={"ID":"0dc94bd1-c2a3-4659-adf9-806287affcc8","Type":"ContainerDied","Data":"4c6bb5cb9736c687d688859ecf410b21278397ce939d50b3422d6430e8663ff8"} Sep 18 21:11:49.787720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:49.787332 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" event={"ID":"0dc94bd1-c2a3-4659-adf9-806287affcc8","Type":"ContainerStarted","Data":"966bcb31df09f96569bc105d53d93c3fce22ab86dbf9963a2983537b492a9cd5"} Sep 18 21:11:49.788879 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:49.788854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" event={"ID":"c438df90-6953-4453-8877-bbf1117bd8f7","Type":"ContainerStarted","Data":"49c30a3fccdbf0cbf715e2aec454f23eac391d05725e2556e0d868415e9014cc"} Sep 18 21:11:52.804640 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:52.804609 2577 generic.go:356] "Generic (PLEG): container finished" podID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerID="07c1c6c66312577d491bcf03c203dc2ea82caf90efe5ba9265a1c87272fa7a71" exitCode=0 Sep 18 21:11:52.805100 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:52.804690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" event={"ID":"0dc94bd1-c2a3-4659-adf9-806287affcc8","Type":"ContainerDied","Data":"07c1c6c66312577d491bcf03c203dc2ea82caf90efe5ba9265a1c87272fa7a71"} Sep 18 21:11:52.806363 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:52.806344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" event={"ID":"c438df90-6953-4453-8877-bbf1117bd8f7","Type":"ContainerStarted","Data":"8db4335f337439c6b0d979a2cb0c28d74b2c7e7a80b6efa4ecc545b3f9f0cb43"} Sep 18 21:11:53.812851 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:53.812811 2577 generic.go:356] "Generic (PLEG): container finished" podID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerID="4db21a09d7934531d7ee8bc08edb65f3a0b0c935680fdbeec167fb09c4d252a4" exitCode=0 Sep 18 21:11:53.813391 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:53.812884 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" event={"ID":"0dc94bd1-c2a3-4659-adf9-806287affcc8","Type":"ContainerDied","Data":"4db21a09d7934531d7ee8bc08edb65f3a0b0c935680fdbeec167fb09c4d252a4"} Sep 18 21:11:53.814531 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:53.814508 2577 generic.go:356] "Generic (PLEG): container finished" podID="c438df90-6953-4453-8877-bbf1117bd8f7" containerID="8db4335f337439c6b0d979a2cb0c28d74b2c7e7a80b6efa4ecc545b3f9f0cb43" exitCode=0 Sep 18 21:11:53.814647 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:53.814590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" event={"ID":"c438df90-6953-4453-8877-bbf1117bd8f7","Type":"ContainerDied","Data":"8db4335f337439c6b0d979a2cb0c28d74b2c7e7a80b6efa4ecc545b3f9f0cb43"} Sep 18 21:11:54.820912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:54.820866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" event={"ID":"c438df90-6953-4453-8877-bbf1117bd8f7","Type":"ContainerStarted","Data":"0161873a1547027ec228f71d8d786b7f9fe457bb675dd5d640ee5afad9bd22d6"} Sep 18 21:11:54.821374 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:54.821143 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:11:54.844947 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:54.844880 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" podStartSLOduration=2.31430304 podStartE2EDuration="6.844858542s" totalImagesPullingTime="3.450037741s" totalInitContainerRuntime="1.080517761s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:54.843107103 +0000 UTC m=+343.772852980" watchObservedRunningTime="2026-09-18 21:11:54.844858542 +0000 UTC m=+343.774604417" Sep 18 21:11:54.943144 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:54.943120 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:55.039062 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.039024 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drm59\" (UniqueName: \"kubernetes.io/projected/0dc94bd1-c2a3-4659-adf9-806287affcc8-kube-api-access-drm59\") pod \"0dc94bd1-c2a3-4659-adf9-806287affcc8\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " Sep 18 21:11:55.039062 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.039069 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-bundle\") pod \"0dc94bd1-c2a3-4659-adf9-806287affcc8\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " Sep 18 21:11:55.039308 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.039150 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-util\") pod \"0dc94bd1-c2a3-4659-adf9-806287affcc8\" (UID: \"0dc94bd1-c2a3-4659-adf9-806287affcc8\") " Sep 18 21:11:55.039865 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.039835 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-bundle" (OuterVolumeSpecName: "bundle") pod "0dc94bd1-c2a3-4659-adf9-806287affcc8" (UID: "0dc94bd1-c2a3-4659-adf9-806287affcc8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:11:55.041226 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.041203 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc94bd1-c2a3-4659-adf9-806287affcc8-kube-api-access-drm59" (OuterVolumeSpecName: "kube-api-access-drm59") pod "0dc94bd1-c2a3-4659-adf9-806287affcc8" (UID: "0dc94bd1-c2a3-4659-adf9-806287affcc8"). InnerVolumeSpecName "kube-api-access-drm59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:11:55.043543 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.043519 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-util" (OuterVolumeSpecName: "util") pod "0dc94bd1-c2a3-4659-adf9-806287affcc8" (UID: "0dc94bd1-c2a3-4659-adf9-806287affcc8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:11:55.140066 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.139981 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-util\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:55.140066 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.140012 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drm59\" (UniqueName: \"kubernetes.io/projected/0dc94bd1-c2a3-4659-adf9-806287affcc8-kube-api-access-drm59\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:55.140066 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.140021 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dc94bd1-c2a3-4659-adf9-806287affcc8-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:11:55.827621 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.827584 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" Sep 18 21:11:55.827621 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.827607 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b052vzt" event={"ID":"0dc94bd1-c2a3-4659-adf9-806287affcc8","Type":"ContainerDied","Data":"966bcb31df09f96569bc105d53d93c3fce22ab86dbf9963a2983537b492a9cd5"} Sep 18 21:11:55.828148 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:11:55.827652 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="966bcb31df09f96569bc105d53d93c3fce22ab86dbf9963a2983537b492a9cd5" Sep 18 21:12:05.830448 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:05.830417 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-sgfsn" Sep 18 21:12:08.460186 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.460150 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k"] Sep 18 21:12:08.460551 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.460456 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerName="util" Sep 18 21:12:08.460551 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.460467 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerName="util" Sep 18 21:12:08.460551 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.460481 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerName="pull" Sep 18 21:12:08.460551 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.460506 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerName="pull" Sep 18 21:12:08.460551 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.460516 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerName="extract" Sep 18 21:12:08.460551 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.460521 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerName="extract" Sep 18 21:12:08.460737 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.460574 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc94bd1-c2a3-4659-adf9-806287affcc8" containerName="extract" Sep 18 21:12:08.466501 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.466482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.471884 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.471845 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 21:12:08.472286 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.472272 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 18 21:12:08.472367 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.472283 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 21:12:08.475017 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.475000 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 18 21:12:08.500515 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.500491 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k"] Sep 18 21:12:08.549176 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.549150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfzh\" (UniqueName: \"kubernetes.io/projected/fded3273-f36c-45d1-b000-03d913f26bd6-kube-api-access-pkfzh\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.549312 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.549183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fded3273-f36c-45d1-b000-03d913f26bd6-tls-certs\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.549312 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.549215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fded3273-f36c-45d1-b000-03d913f26bd6-tmp\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.650683 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.650645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfzh\" (UniqueName: \"kubernetes.io/projected/fded3273-f36c-45d1-b000-03d913f26bd6-kube-api-access-pkfzh\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.650856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.650704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fded3273-f36c-45d1-b000-03d913f26bd6-tls-certs\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.650856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.650757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fded3273-f36c-45d1-b000-03d913f26bd6-tmp\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.653062 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.653040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fded3273-f36c-45d1-b000-03d913f26bd6-tmp\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.653288 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.653267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fded3273-f36c-45d1-b000-03d913f26bd6-tls-certs\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.670376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.670346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfzh\" (UniqueName: \"kubernetes.io/projected/fded3273-f36c-45d1-b000-03d913f26bd6-kube-api-access-pkfzh\") pod \"kube-auth-proxy-85cdd48bb5-v6c8k\" (UID: \"fded3273-f36c-45d1-b000-03d913f26bd6\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:08.775803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:08.775724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" Sep 18 21:12:09.079471 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:09.079443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k"] Sep 18 21:12:09.081633 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:12:09.081599 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfded3273_f36c_45d1_b000_03d913f26bd6.slice/crio-f5d04bc9e60ee75ce8cd251517472ae8bd22236eb42778632c97a5e0ff055926 WatchSource:0}: Error finding container f5d04bc9e60ee75ce8cd251517472ae8bd22236eb42778632c97a5e0ff055926: Status 404 returned error can't find the container with id f5d04bc9e60ee75ce8cd251517472ae8bd22236eb42778632c97a5e0ff055926 Sep 18 21:12:09.902294 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:09.902212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" event={"ID":"fded3273-f36c-45d1-b000-03d913f26bd6","Type":"ContainerStarted","Data":"f5d04bc9e60ee75ce8cd251517472ae8bd22236eb42778632c97a5e0ff055926"} Sep 18 21:12:10.661900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.661861 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb"] Sep 18 21:12:10.670847 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.670814 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.673808 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.673643 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 18 21:12:10.673808 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.673676 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 18 21:12:10.673808 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.673747 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jqqqw\"" Sep 18 21:12:10.674115 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.674061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 18 21:12:10.674499 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.674337 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 18 21:12:10.690557 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.690526 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb"] Sep 18 21:12:10.770342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.770310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8hl\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-kube-api-access-kt8hl\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.770512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.770350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.770512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.770390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.770512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.770452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.770512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.770500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.770691 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.770537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77b1e1-8f49-43ff-9367-4fe104f50853-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.770691 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.770567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.872090 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.872052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8hl\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-kube-api-access-kt8hl\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.872283 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.872104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.872283 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.872149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.872283 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.872183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.872283 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.872226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.872283 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.872260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77b1e1-8f49-43ff-9367-4fe104f50853-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.872538 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.872301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.873158 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.873105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.875131 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.875106 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77b1e1-8f49-43ff-9367-4fe104f50853-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.875238 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.875129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.875238 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.875135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.875318 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.875282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.880964 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.880909 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.881199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.881177 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8hl\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-kube-api-access-kt8hl\") pod \"istiod-openshift-gateway-7b7995d5cd-n8jcb\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:10.983708 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:10.983616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:11.914859 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:11.914822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" event={"ID":"fded3273-f36c-45d1-b000-03d913f26bd6","Type":"ContainerStarted","Data":"c0724fad7f0dc70e4d6ffe50c9f273b8429f47899a11b74af5851dbc2de10e47"} Sep 18 21:12:11.934336 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:11.934289 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-v6c8k" podStartSLOduration=1.2151895129999999 podStartE2EDuration="3.934275616s" totalImagesPullingTime="2.719086103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:12:11.932526185 +0000 UTC m=+360.862272081" watchObservedRunningTime="2026-09-18 21:12:11.934275616 +0000 UTC m=+360.864021488" Sep 18 21:12:12.018408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:12.018383 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb"] Sep 18 21:12:12.020183 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:12:12.020154 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd77b1e1_8f49_43ff_9367_4fe104f50853.slice/crio-6454ac0b81f36291193d0b6d59ef61640ce0c99fa8d4f359648ed83169f1962b WatchSource:0}: Error finding container 6454ac0b81f36291193d0b6d59ef61640ce0c99fa8d4f359648ed83169f1962b: Status 404 returned error can't find the container with id 6454ac0b81f36291193d0b6d59ef61640ce0c99fa8d4f359648ed83169f1962b Sep 18 21:12:12.923189 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:12.923154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" event={"ID":"bd77b1e1-8f49-43ff-9367-4fe104f50853","Type":"ContainerStarted","Data":"6454ac0b81f36291193d0b6d59ef61640ce0c99fa8d4f359648ed83169f1962b"} Sep 18 21:12:14.426774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:14.426737 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:12:14.427049 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:14.426804 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:12:14.935089 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:14.935041 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" event={"ID":"bd77b1e1-8f49-43ff-9367-4fe104f50853","Type":"ContainerStarted","Data":"8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93"} Sep 18 21:12:14.935281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:14.935194 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:15.942424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:15.942388 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:15.950559 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:15.950511 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" podStartSLOduration=3.5462822000000003 podStartE2EDuration="5.950493799s" totalImagesPullingTime="2.404211599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:12:14.955716051 +0000 UTC m=+363.885461917" watchObservedRunningTime="2026-09-18 21:12:15.950493799 +0000 UTC m=+364.880239692" Sep 18 21:12:15.951314 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:15.951290 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g"] Sep 18 21:12:15.954803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:15.954784 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:15.957410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:15.957391 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rgbls\"" Sep 18 21:12:15.968692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:15.968662 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g"] Sep 18 21:12:16.120375 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.120599 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.120599 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.120599 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.120720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkw5\" (UniqueName: \"kubernetes.io/projected/26452a17-3521-44a0-88fa-9f1a9cd30684-kube-api-access-6jkw5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.120720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.120720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.120847 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120726 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26452a17-3521-44a0-88fa-9f1a9cd30684-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.120847 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.120790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.223949 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.223854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.223949 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.223923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.223979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkw5\" (UniqueName: \"kubernetes.io/projected/26452a17-3521-44a0-88fa-9f1a9cd30684-kube-api-access-6jkw5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26452a17-3521-44a0-88fa-9f1a9cd30684-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224558 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224558 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224558 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.224993 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.224943 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26452a17-3521-44a0-88fa-9f1a9cd30684-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.226431 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.226411 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.226870 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.226850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.232178 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.232155 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26452a17-3521-44a0-88fa-9f1a9cd30684-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.232261 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.232188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkw5\" (UniqueName: \"kubernetes.io/projected/26452a17-3521-44a0-88fa-9f1a9cd30684-kube-api-access-6jkw5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g\" (UID: \"26452a17-3521-44a0-88fa-9f1a9cd30684\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.265250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.265208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:16.545811 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.545741 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g"] Sep 18 21:12:16.547996 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:12:16.547966 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26452a17_3521_44a0_88fa_9f1a9cd30684.slice/crio-1d1b9e4e93ad62cf56894e834f0dda1d861149388825e0c19cae15950e1f91ad WatchSource:0}: Error finding container 1d1b9e4e93ad62cf56894e834f0dda1d861149388825e0c19cae15950e1f91ad: Status 404 returned error can't find the container with id 1d1b9e4e93ad62cf56894e834f0dda1d861149388825e0c19cae15950e1f91ad Sep 18 21:12:16.946775 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:16.946741 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" event={"ID":"26452a17-3521-44a0-88fa-9f1a9cd30684","Type":"ContainerStarted","Data":"1d1b9e4e93ad62cf56894e834f0dda1d861149388825e0c19cae15950e1f91ad"} Sep 18 21:12:19.900295 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:19.900258 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:12:19.900718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:19.900363 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:12:19.900718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:19.900408 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:12:20.968378 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:20.968334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" event={"ID":"26452a17-3521-44a0-88fa-9f1a9cd30684","Type":"ContainerStarted","Data":"5469c21b6f2054d80f01e9801834dd3f6b9abab5b867b8c881130dee59e3495c"} Sep 18 21:12:21.008780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:21.008724 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" podStartSLOduration=2.6590005679999997 podStartE2EDuration="6.00870874s" totalImagesPullingTime="3.349708172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:12:21.005880453 +0000 UTC m=+369.935626326" watchObservedRunningTime="2026-09-18 21:12:21.00870874 +0000 UTC m=+369.938454612" Sep 18 21:12:21.266520 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:21.266422 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:21.270983 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:21.270936 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:21.974346 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:21.974316 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:21.975292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:21.975275 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g" Sep 18 21:12:29.307444 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.307409 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d54fj"] Sep 18 21:12:29.310437 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.310417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:29.313043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.313023 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7sh66\"" Sep 18 21:12:29.313140 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.313022 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 18 21:12:29.325934 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.325912 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74"] Sep 18 21:12:29.329135 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.329117 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d54fj"] Sep 18 21:12:29.329220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.329197 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:29.331646 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.331626 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 18 21:12:29.331811 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.331793 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8mgrb\"" Sep 18 21:12:29.332761 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.332739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhvw\" (UniqueName: \"kubernetes.io/projected/fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d-kube-api-access-gbhvw\") pod \"kserve-controller-manager-d6f4b65b8-d54fj\" (UID: \"fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:29.333040 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.333006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d-cert\") pod \"kserve-controller-manager-d6f4b65b8-d54fj\" (UID: \"fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:29.346321 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.346295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74"] Sep 18 21:12:29.433803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.433768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d-cert\") pod \"kserve-controller-manager-d6f4b65b8-d54fj\" (UID: \"fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:29.434016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.433843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a424b6e3-ada2-49a5-9191-457d24496f54-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-snz74\" (UID: \"a424b6e3-ada2-49a5-9191-457d24496f54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:29.434016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.433995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djwhm\" (UniqueName: \"kubernetes.io/projected/a424b6e3-ada2-49a5-9191-457d24496f54-kube-api-access-djwhm\") pod \"llmisvc-controller-manager-777c6bd7d5-snz74\" (UID: \"a424b6e3-ada2-49a5-9191-457d24496f54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:29.434147 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.434098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhvw\" (UniqueName: \"kubernetes.io/projected/fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d-kube-api-access-gbhvw\") pod \"kserve-controller-manager-d6f4b65b8-d54fj\" (UID: \"fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:29.436477 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.436452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d-cert\") pod \"kserve-controller-manager-d6f4b65b8-d54fj\" (UID: \"fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:29.442447 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.442425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhvw\" (UniqueName: \"kubernetes.io/projected/fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d-kube-api-access-gbhvw\") pod \"kserve-controller-manager-d6f4b65b8-d54fj\" (UID: \"fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:29.535106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.535072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a424b6e3-ada2-49a5-9191-457d24496f54-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-snz74\" (UID: \"a424b6e3-ada2-49a5-9191-457d24496f54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:29.535106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.535108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djwhm\" (UniqueName: \"kubernetes.io/projected/a424b6e3-ada2-49a5-9191-457d24496f54-kube-api-access-djwhm\") pod \"llmisvc-controller-manager-777c6bd7d5-snz74\" (UID: \"a424b6e3-ada2-49a5-9191-457d24496f54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:29.535299 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:12:29.535204 2577 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 18 21:12:29.535299 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:12:29.535287 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a424b6e3-ada2-49a5-9191-457d24496f54-cert podName:a424b6e3-ada2-49a5-9191-457d24496f54 nodeName:}" failed. No retries permitted until 2026-09-18 21:12:30.035266984 +0000 UTC m=+378.965012849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a424b6e3-ada2-49a5-9191-457d24496f54-cert") pod "llmisvc-controller-manager-777c6bd7d5-snz74" (UID: "a424b6e3-ada2-49a5-9191-457d24496f54") : secret "llmisvc-webhook-server-cert" not found Sep 18 21:12:29.544420 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.544397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djwhm\" (UniqueName: \"kubernetes.io/projected/a424b6e3-ada2-49a5-9191-457d24496f54-kube-api-access-djwhm\") pod \"llmisvc-controller-manager-777c6bd7d5-snz74\" (UID: \"a424b6e3-ada2-49a5-9191-457d24496f54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:29.619644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.619617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:29.908324 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:29.908296 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d54fj"] Sep 18 21:12:29.910243 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:12:29.910215 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2eaf8b_1d10_4e13_82ca_6743faa0eb9d.slice/crio-eebda7d015ea0928d42630a8b524eaf94dfedb104cb7d52a31a192d9b8c5df94 WatchSource:0}: Error finding container eebda7d015ea0928d42630a8b524eaf94dfedb104cb7d52a31a192d9b8c5df94: Status 404 returned error can't find the container with id eebda7d015ea0928d42630a8b524eaf94dfedb104cb7d52a31a192d9b8c5df94 Sep 18 21:12:30.010093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.010055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" event={"ID":"fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d","Type":"ContainerStarted","Data":"eebda7d015ea0928d42630a8b524eaf94dfedb104cb7d52a31a192d9b8c5df94"} Sep 18 21:12:30.038446 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.038416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a424b6e3-ada2-49a5-9191-457d24496f54-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-snz74\" (UID: \"a424b6e3-ada2-49a5-9191-457d24496f54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:30.040789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.040762 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a424b6e3-ada2-49a5-9191-457d24496f54-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-snz74\" (UID: \"a424b6e3-ada2-49a5-9191-457d24496f54\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:30.150354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.150316 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb"] Sep 18 21:12:30.150564 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.150541 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" podUID="bd77b1e1-8f49-43ff-9367-4fe104f50853" containerName="discovery" containerID="cri-o://8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93" gracePeriod=30 Sep 18 21:12:30.238767 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.238727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:30.388939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.388912 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:30.441078 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.440979 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-kubeconfig\") pod \"bd77b1e1-8f49-43ff-9367-4fe104f50853\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " Sep 18 21:12:30.441078 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.441026 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-ca-configmap\") pod \"bd77b1e1-8f49-43ff-9367-4fe104f50853\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " Sep 18 21:12:30.441078 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.441063 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-dns-cert\") pod \"bd77b1e1-8f49-43ff-9367-4fe104f50853\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " Sep 18 21:12:30.441367 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.441128 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt8hl\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-kube-api-access-kt8hl\") pod \"bd77b1e1-8f49-43ff-9367-4fe104f50853\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " Sep 18 21:12:30.441367 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.441164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-token\") pod \"bd77b1e1-8f49-43ff-9367-4fe104f50853\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " Sep 18 21:12:30.441367 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.441197 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-cacerts\") pod \"bd77b1e1-8f49-43ff-9367-4fe104f50853\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " Sep 18 21:12:30.441367 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.441236 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77b1e1-8f49-43ff-9367-4fe104f50853-local-certs\") pod \"bd77b1e1-8f49-43ff-9367-4fe104f50853\" (UID: \"bd77b1e1-8f49-43ff-9367-4fe104f50853\") " Sep 18 21:12:30.441579 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.441548 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "bd77b1e1-8f49-43ff-9367-4fe104f50853" (UID: "bd77b1e1-8f49-43ff-9367-4fe104f50853"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:12:30.444166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.444124 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "bd77b1e1-8f49-43ff-9367-4fe104f50853" (UID: "bd77b1e1-8f49-43ff-9367-4fe104f50853"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:12:30.444166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.444129 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd77b1e1-8f49-43ff-9367-4fe104f50853-local-certs" (OuterVolumeSpecName: "local-certs") pod "bd77b1e1-8f49-43ff-9367-4fe104f50853" (UID: "bd77b1e1-8f49-43ff-9367-4fe104f50853"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:12:30.444372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.444227 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-cacerts" (OuterVolumeSpecName: "cacerts") pod "bd77b1e1-8f49-43ff-9367-4fe104f50853" (UID: "bd77b1e1-8f49-43ff-9367-4fe104f50853"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:12:30.444372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.444279 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "bd77b1e1-8f49-43ff-9367-4fe104f50853" (UID: "bd77b1e1-8f49-43ff-9367-4fe104f50853"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:12:30.444484 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.444386 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-kube-api-access-kt8hl" (OuterVolumeSpecName: "kube-api-access-kt8hl") pod "bd77b1e1-8f49-43ff-9367-4fe104f50853" (UID: "bd77b1e1-8f49-43ff-9367-4fe104f50853"). InnerVolumeSpecName "kube-api-access-kt8hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:12:30.444625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.444602 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd77b1e1-8f49-43ff-9367-4fe104f50853" (UID: "bd77b1e1-8f49-43ff-9367-4fe104f50853"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:12:30.542622 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.542588 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:12:30.542622 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.542619 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-cacerts\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:12:30.542622 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.542630 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd77b1e1-8f49-43ff-9367-4fe104f50853-local-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:12:30.543093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.542641 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-kubeconfig\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:12:30.543093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.542654 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-ca-configmap\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:12:30.543093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.542667 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bd77b1e1-8f49-43ff-9367-4fe104f50853-istio-csr-dns-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:12:30.543093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.542678 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8hl\" (UniqueName: \"kubernetes.io/projected/bd77b1e1-8f49-43ff-9367-4fe104f50853-kube-api-access-kt8hl\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:12:30.543981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:30.543896 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74"] Sep 18 21:12:30.546337 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:12:30.546307 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda424b6e3_ada2_49a5_9191_457d24496f54.slice/crio-4bcb539f452fa466691f90f3242ef06eb46823f1e994f9a7af95ef24a3f3b5d9 WatchSource:0}: Error finding container 4bcb539f452fa466691f90f3242ef06eb46823f1e994f9a7af95ef24a3f3b5d9: Status 404 returned error can't find the container with id 4bcb539f452fa466691f90f3242ef06eb46823f1e994f9a7af95ef24a3f3b5d9 Sep 18 21:12:31.017896 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.017857 2577 generic.go:356] "Generic (PLEG): container finished" podID="bd77b1e1-8f49-43ff-9367-4fe104f50853" containerID="8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93" exitCode=0 Sep 18 21:12:31.018106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.017903 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" Sep 18 21:12:31.018106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.017968 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" event={"ID":"bd77b1e1-8f49-43ff-9367-4fe104f50853","Type":"ContainerDied","Data":"8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93"} Sep 18 21:12:31.018106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.018037 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb" event={"ID":"bd77b1e1-8f49-43ff-9367-4fe104f50853","Type":"ContainerDied","Data":"6454ac0b81f36291193d0b6d59ef61640ce0c99fa8d4f359648ed83169f1962b"} Sep 18 21:12:31.018106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.018069 2577 scope.go:117] "RemoveContainer" containerID="8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93" Sep 18 21:12:31.019707 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.019601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" event={"ID":"a424b6e3-ada2-49a5-9191-457d24496f54","Type":"ContainerStarted","Data":"4bcb539f452fa466691f90f3242ef06eb46823f1e994f9a7af95ef24a3f3b5d9"} Sep 18 21:12:31.028766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.028561 2577 scope.go:117] "RemoveContainer" containerID="8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93" Sep 18 21:12:31.029100 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:12:31.029074 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93\": container with ID starting with 8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93 not found: ID does not exist" containerID="8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93" Sep 18 21:12:31.029198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.029113 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93"} err="failed to get container status \"8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93\": rpc error: code = NotFound desc = could not find container \"8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93\": container with ID starting with 8ed9b7fd32784a899b7e028cdba37c396d6e571a17f148e39babf5c26bc1bf93 not found: ID does not exist" Sep 18 21:12:31.059357 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.059324 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb"] Sep 18 21:12:31.063581 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.063553 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n8jcb"] Sep 18 21:12:31.359472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:31.359441 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd77b1e1-8f49-43ff-9367-4fe104f50853" path="/var/lib/kubelet/pods/bd77b1e1-8f49-43ff-9367-4fe104f50853/volumes" Sep 18 21:12:34.040218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:34.040183 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" event={"ID":"fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d","Type":"ContainerStarted","Data":"84715c25cfc560bf733901c03d83feeba1578c2b93c544bda128af19377f8e92"} Sep 18 21:12:34.040670 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:34.040304 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:12:34.041888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:34.041860 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" event={"ID":"a424b6e3-ada2-49a5-9191-457d24496f54","Type":"ContainerStarted","Data":"f9b60ae8a83cc1673cc4fbd7a2913dd3ff4d1460129152608b9b349c24793833"} Sep 18 21:12:34.042020 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:34.041992 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:12:34.073037 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:34.072988 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" podStartSLOduration=2.355503314 podStartE2EDuration="5.072947945s" totalImagesPullingTime="2.717444631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:12:34.067459712 +0000 UTC m=+382.997205586" watchObservedRunningTime="2026-09-18 21:12:34.072947945 +0000 UTC m=+383.002693818" Sep 18 21:12:34.126086 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:12:34.126032 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" podStartSLOduration=2.345934544 podStartE2EDuration="5.126016285s" totalImagesPullingTime="2.780081741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:12:34.119543585 +0000 UTC m=+383.049289459" watchObservedRunningTime="2026-09-18 21:12:34.126016285 +0000 UTC m=+383.055762168" Sep 18 21:13:05.049007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:05.048949 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-snz74" Sep 18 21:13:05.052018 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:05.051999 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d54fj" Sep 18 21:13:12.864585 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:12.864550 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ddcb54c4b-8qgd4"] Sep 18 21:13:12.864971 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:12.864883 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd77b1e1-8f49-43ff-9367-4fe104f50853" containerName="discovery" Sep 18 21:13:12.864971 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:12.864893 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd77b1e1-8f49-43ff-9367-4fe104f50853" containerName="discovery" Sep 18 21:13:12.865053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:12.865004 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd77b1e1-8f49-43ff-9367-4fe104f50853" containerName="discovery" Sep 18 21:13:12.869220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:12.869200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:12.890090 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:12.890061 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ddcb54c4b-8qgd4"] Sep 18 21:13:13.002614 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.002584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fb4badf-4634-4af6-b0ab-a55f38647996-console-serving-cert\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.002778 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.002622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vzwp\" (UniqueName: \"kubernetes.io/projected/2fb4badf-4634-4af6-b0ab-a55f38647996-kube-api-access-4vzwp\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.002778 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.002731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-console-config\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.002858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.002789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-trusted-ca-bundle\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.002858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.002813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fb4badf-4634-4af6-b0ab-a55f38647996-console-oauth-config\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.002858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.002830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-service-ca\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.002975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.002888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-oauth-serving-cert\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.103995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.103938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-console-config\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.103995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.103998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-trusted-ca-bundle\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fb4badf-4634-4af6-b0ab-a55f38647996-console-oauth-config\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-service-ca\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-oauth-serving-cert\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104091 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fb4badf-4634-4af6-b0ab-a55f38647996-console-serving-cert\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vzwp\" (UniqueName: \"kubernetes.io/projected/2fb4badf-4634-4af6-b0ab-a55f38647996-kube-api-access-4vzwp\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104812 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-console-config\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104922 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104843 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-oauth-serving-cert\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-service-ca\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.104999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.104934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb4badf-4634-4af6-b0ab-a55f38647996-trusted-ca-bundle\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.106666 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.106646 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fb4badf-4634-4af6-b0ab-a55f38647996-console-oauth-config\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.106751 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.106741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fb4badf-4634-4af6-b0ab-a55f38647996-console-serving-cert\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.112832 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.112808 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vzwp\" (UniqueName: \"kubernetes.io/projected/2fb4badf-4634-4af6-b0ab-a55f38647996-kube-api-access-4vzwp\") pod \"console-7ddcb54c4b-8qgd4\" (UID: \"2fb4badf-4634-4af6-b0ab-a55f38647996\") " pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.178054 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.177972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:13.450933 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:13.450860 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ddcb54c4b-8qgd4"] Sep 18 21:13:13.453859 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:13.453824 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb4badf_4634_4af6_b0ab_a55f38647996.slice/crio-b17912834689c7c369b2ba028e0d0d482d840de5cf110b34e370f82b1fe2a9ed WatchSource:0}: Error finding container b17912834689c7c369b2ba028e0d0d482d840de5cf110b34e370f82b1fe2a9ed: Status 404 returned error can't find the container with id b17912834689c7c369b2ba028e0d0d482d840de5cf110b34e370f82b1fe2a9ed Sep 18 21:13:14.241964 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:14.241924 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ddcb54c4b-8qgd4" event={"ID":"2fb4badf-4634-4af6-b0ab-a55f38647996","Type":"ContainerStarted","Data":"214b3b0290d813082dac0b056258d2085f30366b70767b78c5a69c2d7eb7050c"} Sep 18 21:13:14.242330 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:14.242027 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ddcb54c4b-8qgd4" event={"ID":"2fb4badf-4634-4af6-b0ab-a55f38647996","Type":"ContainerStarted","Data":"b17912834689c7c369b2ba028e0d0d482d840de5cf110b34e370f82b1fe2a9ed"} Sep 18 21:13:14.260416 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:14.260367 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7ddcb54c4b-8qgd4" podStartSLOduration=2.260352388 podStartE2EDuration="2.260352388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:13:14.259222974 +0000 UTC m=+423.188968846" watchObservedRunningTime="2026-09-18 21:13:14.260352388 +0000 UTC m=+423.190098262" Sep 18 21:13:23.178141 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:23.178096 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:23.178626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:23.178166 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:23.182843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:23.182821 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:23.288635 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:23.288606 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ddcb54c4b-8qgd4" Sep 18 21:13:23.431098 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:23.431024 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f6f8797c8-w8wmq"] Sep 18 21:13:29.667715 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.667628 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq"] Sep 18 21:13:29.670058 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.670041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.679231 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.679210 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 21:13:29.679385 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.679370 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 21:13:29.680101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.680087 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvq7g\"" Sep 18 21:13:29.691441 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.691413 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq"] Sep 18 21:13:29.754456 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.754419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.754634 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.754494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hhj\" (UniqueName: \"kubernetes.io/projected/12ec0ce3-c231-4df7-9abf-0408ff69f947-kube-api-access-x5hhj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.754634 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.754523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.797070 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.797040 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx"] Sep 18 21:13:29.799585 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.799569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.818385 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.818360 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx"] Sep 18 21:13:29.855214 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.855177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mts\" (UniqueName: \"kubernetes.io/projected/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-kube-api-access-58mts\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.855408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.855226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hhj\" (UniqueName: \"kubernetes.io/projected/12ec0ce3-c231-4df7-9abf-0408ff69f947-kube-api-access-x5hhj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.855408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.855286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.855408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.855317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.855408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.855386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.855583 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.855413 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.855699 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.855679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.855734 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.855683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.868495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.868461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hhj\" (UniqueName: \"kubernetes.io/projected/12ec0ce3-c231-4df7-9abf-0408ff69f947-kube-api-access-x5hhj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.955243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.955165 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h"] Sep 18 21:13:29.956290 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.956271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mts\" (UniqueName: \"kubernetes.io/projected/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-kube-api-access-58mts\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.956365 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.956318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.956405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.956367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.956719 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.956703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.956805 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.956738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.957824 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.957809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:29.971647 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.971614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mts\" (UniqueName: \"kubernetes.io/projected/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-kube-api-access-58mts\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:29.979107 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.979079 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:29.981137 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:29.981111 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h"] Sep 18 21:13:30.057369 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.057333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbslm\" (UniqueName: \"kubernetes.io/projected/5e8684e6-c38c-4482-9485-c91770475fa9-kube-api-access-gbslm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.057519 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.057414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.057519 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.057467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.096052 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.096017 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl"] Sep 18 21:13:30.098641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.098624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.108339 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.108320 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:30.118724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.118694 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl"] Sep 18 21:13:30.158373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.158220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.158373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.158360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbslm\" (UniqueName: \"kubernetes.io/projected/5e8684e6-c38c-4482-9485-c91770475fa9-kube-api-access-gbslm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.158516 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.158404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz24g\" (UniqueName: \"kubernetes.io/projected/ed012a84-b001-4a5e-b019-cfdf271ef183-kube-api-access-sz24g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.158516 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.158442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.158516 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.158480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.158675 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.158531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.159080 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.158985 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.159080 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.159032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.180536 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.180505 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbslm\" (UniqueName: \"kubernetes.io/projected/5e8684e6-c38c-4482-9485-c91770475fa9-kube-api-access-gbslm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.259597 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.259506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz24g\" (UniqueName: \"kubernetes.io/projected/ed012a84-b001-4a5e-b019-cfdf271ef183-kube-api-access-sz24g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.259597 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.259565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.259793 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.259606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.259985 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.259946 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.260034 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.260003 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.267575 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.267543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:30.290372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.290344 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz24g\" (UniqueName: \"kubernetes.io/projected/ed012a84-b001-4a5e-b019-cfdf271ef183-kube-api-access-sz24g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.293537 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.293499 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq"] Sep 18 21:13:30.297255 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:30.297229 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ec0ce3_c231_4df7_9abf_0408ff69f947.slice/crio-3bbf08ee001228d6b5c74fe5ab9a5e7e4b3284b216d6439e627c72755cc86f73 WatchSource:0}: Error finding container 3bbf08ee001228d6b5c74fe5ab9a5e7e4b3284b216d6439e627c72755cc86f73: Status 404 returned error can't find the container with id 3bbf08ee001228d6b5c74fe5ab9a5e7e4b3284b216d6439e627c72755cc86f73 Sep 18 21:13:30.326210 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.326179 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" event={"ID":"12ec0ce3-c231-4df7-9abf-0408ff69f947","Type":"ContainerStarted","Data":"3bbf08ee001228d6b5c74fe5ab9a5e7e4b3284b216d6439e627c72755cc86f73"} Sep 18 21:13:30.392500 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.392118 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx"] Sep 18 21:13:30.408378 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.408336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:30.436099 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:30.436056 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e30e51a_626c_4213_ba4d_9f84ed3b8b48.slice/crio-683dfc985c1050c59cf0be7a9f5f0d38d7c40cad60e6ff57029553cab91fd5e2 WatchSource:0}: Error finding container 683dfc985c1050c59cf0be7a9f5f0d38d7c40cad60e6ff57029553cab91fd5e2: Status 404 returned error can't find the container with id 683dfc985c1050c59cf0be7a9f5f0d38d7c40cad60e6ff57029553cab91fd5e2 Sep 18 21:13:30.545916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.545884 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h"] Sep 18 21:13:30.593023 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:30.592987 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8684e6_c38c_4482_9485_c91770475fa9.slice/crio-a7513f133a9021a28b2ff7c750c3d8b6303bb784b2904dcb4034bc406ab1b55d WatchSource:0}: Error finding container a7513f133a9021a28b2ff7c750c3d8b6303bb784b2904dcb4034bc406ab1b55d: Status 404 returned error can't find the container with id a7513f133a9021a28b2ff7c750c3d8b6303bb784b2904dcb4034bc406ab1b55d Sep 18 21:13:30.692923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:30.692894 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl"] Sep 18 21:13:30.695755 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:30.695724 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded012a84_b001_4a5e_b019_cfdf271ef183.slice/crio-8b60ebb786eb926d5bf8e9c5f67084d5ca04652a29bd29ee29bdb918c7d24feb WatchSource:0}: Error finding container 8b60ebb786eb926d5bf8e9c5f67084d5ca04652a29bd29ee29bdb918c7d24feb: Status 404 returned error can't find the container with id 8b60ebb786eb926d5bf8e9c5f67084d5ca04652a29bd29ee29bdb918c7d24feb Sep 18 21:13:31.332805 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.332772 2577 generic.go:356] "Generic (PLEG): container finished" podID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerID="944affdf7123d1be2007a19fd7246643752b53781d4266a23a62e319e1110722" exitCode=0 Sep 18 21:13:31.333015 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.332865 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" event={"ID":"12ec0ce3-c231-4df7-9abf-0408ff69f947","Type":"ContainerDied","Data":"944affdf7123d1be2007a19fd7246643752b53781d4266a23a62e319e1110722"} Sep 18 21:13:31.334490 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.334468 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e8684e6-c38c-4482-9485-c91770475fa9" containerID="6b912ddb9a4cd7ca0a4afc0c4b697ef0bcc9948a608ffd15261b7403e5f85a03" exitCode=0 Sep 18 21:13:31.334582 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.334557 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" event={"ID":"5e8684e6-c38c-4482-9485-c91770475fa9","Type":"ContainerDied","Data":"6b912ddb9a4cd7ca0a4afc0c4b697ef0bcc9948a608ffd15261b7403e5f85a03"} Sep 18 21:13:31.334644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.334610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" event={"ID":"5e8684e6-c38c-4482-9485-c91770475fa9","Type":"ContainerStarted","Data":"a7513f133a9021a28b2ff7c750c3d8b6303bb784b2904dcb4034bc406ab1b55d"} Sep 18 21:13:31.336319 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.336299 2577 generic.go:356] "Generic (PLEG): container finished" podID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerID="f57d755038ab405fd1275dc07d938a9038e4b39469fb7d02b251a3e125c4c22d" exitCode=0 Sep 18 21:13:31.336403 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.336385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" event={"ID":"7e30e51a-626c-4213-ba4d-9f84ed3b8b48","Type":"ContainerDied","Data":"f57d755038ab405fd1275dc07d938a9038e4b39469fb7d02b251a3e125c4c22d"} Sep 18 21:13:31.336467 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.336429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" event={"ID":"7e30e51a-626c-4213-ba4d-9f84ed3b8b48","Type":"ContainerStarted","Data":"683dfc985c1050c59cf0be7a9f5f0d38d7c40cad60e6ff57029553cab91fd5e2"} Sep 18 21:13:31.338266 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.338242 2577 generic.go:356] "Generic (PLEG): container finished" podID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerID="8a7d934b1b578390e7bc5eb0d2b527b988f03cc7159dda8a66fa1900f9df16d8" exitCode=0 Sep 18 21:13:31.338364 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.338284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" event={"ID":"ed012a84-b001-4a5e-b019-cfdf271ef183","Type":"ContainerDied","Data":"8a7d934b1b578390e7bc5eb0d2b527b988f03cc7159dda8a66fa1900f9df16d8"} Sep 18 21:13:31.338364 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:31.338321 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" event={"ID":"ed012a84-b001-4a5e-b019-cfdf271ef183","Type":"ContainerStarted","Data":"8b60ebb786eb926d5bf8e9c5f67084d5ca04652a29bd29ee29bdb918c7d24feb"} Sep 18 21:13:32.345525 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:32.345481 2577 generic.go:356] "Generic (PLEG): container finished" podID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerID="4cf7e3625dfcae8d35a4f2bfc3b74b80d907975e4d93868c79801c2d91486faa" exitCode=0 Sep 18 21:13:32.346005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:32.345563 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" event={"ID":"7e30e51a-626c-4213-ba4d-9f84ed3b8b48","Type":"ContainerDied","Data":"4cf7e3625dfcae8d35a4f2bfc3b74b80d907975e4d93868c79801c2d91486faa"} Sep 18 21:13:32.347799 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:32.347773 2577 generic.go:356] "Generic (PLEG): container finished" podID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerID="1b8a92ec2f6b9f011f8ebb84044d5a372af7e7bcc72e119794e6d1b21a19ff17" exitCode=0 Sep 18 21:13:32.347884 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:32.347809 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" event={"ID":"ed012a84-b001-4a5e-b019-cfdf271ef183","Type":"ContainerDied","Data":"1b8a92ec2f6b9f011f8ebb84044d5a372af7e7bcc72e119794e6d1b21a19ff17"} Sep 18 21:13:33.356769 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:33.356732 2577 generic.go:356] "Generic (PLEG): container finished" podID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerID="99b95108d66d94173a5d45be5bbcba341b77e1d6e97d8c6c3ac397b0f76deb4a" exitCode=0 Sep 18 21:13:33.358438 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:33.358407 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" event={"ID":"7e30e51a-626c-4213-ba4d-9f84ed3b8b48","Type":"ContainerDied","Data":"99b95108d66d94173a5d45be5bbcba341b77e1d6e97d8c6c3ac397b0f76deb4a"} Sep 18 21:13:33.359098 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:33.359077 2577 generic.go:356] "Generic (PLEG): container finished" podID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerID="8b4a9b2ccf317afab572d933d6a86712d310654c47f2fa50def396949a4ef969" exitCode=0 Sep 18 21:13:33.359205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:33.359160 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" event={"ID":"ed012a84-b001-4a5e-b019-cfdf271ef183","Type":"ContainerDied","Data":"8b4a9b2ccf317afab572d933d6a86712d310654c47f2fa50def396949a4ef969"} Sep 18 21:13:33.360916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:33.360893 2577 generic.go:356] "Generic (PLEG): container finished" podID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerID="290dc90e91ead96e44cae3b014e97428969ec918e83423494451d1052ec91096" exitCode=0 Sep 18 21:13:33.361014 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:33.360984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" event={"ID":"12ec0ce3-c231-4df7-9abf-0408ff69f947","Type":"ContainerDied","Data":"290dc90e91ead96e44cae3b014e97428969ec918e83423494451d1052ec91096"} Sep 18 21:13:33.362870 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:33.362836 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e8684e6-c38c-4482-9485-c91770475fa9" containerID="928ca60f4d04cbfd595f5d8ddd57af030f84e1a722e994879e607ad958e511ae" exitCode=0 Sep 18 21:13:33.363006 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:33.362875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" event={"ID":"5e8684e6-c38c-4482-9485-c91770475fa9","Type":"ContainerDied","Data":"928ca60f4d04cbfd595f5d8ddd57af030f84e1a722e994879e607ad958e511ae"} Sep 18 21:13:34.370365 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.370333 2577 generic.go:356] "Generic (PLEG): container finished" podID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerID="4383751f1119a1301ebef642d2d13ba0c84a82b6f8cdf4310181e729fc396b22" exitCode=0 Sep 18 21:13:34.370782 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.370425 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" event={"ID":"12ec0ce3-c231-4df7-9abf-0408ff69f947","Type":"ContainerDied","Data":"4383751f1119a1301ebef642d2d13ba0c84a82b6f8cdf4310181e729fc396b22"} Sep 18 21:13:34.372381 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.372360 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e8684e6-c38c-4482-9485-c91770475fa9" containerID="bea175f4db63a3c2aac867b8e6487d14eade92eecfa15b2216ce12d9f242311f" exitCode=0 Sep 18 21:13:34.372485 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.372441 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" event={"ID":"5e8684e6-c38c-4482-9485-c91770475fa9","Type":"ContainerDied","Data":"bea175f4db63a3c2aac867b8e6487d14eade92eecfa15b2216ce12d9f242311f"} Sep 18 21:13:34.503444 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.503418 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:34.531094 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.531065 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:34.604572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.604540 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz24g\" (UniqueName: \"kubernetes.io/projected/ed012a84-b001-4a5e-b019-cfdf271ef183-kube-api-access-sz24g\") pod \"ed012a84-b001-4a5e-b019-cfdf271ef183\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " Sep 18 21:13:34.604772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.604586 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-bundle\") pod \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " Sep 18 21:13:34.604772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.604655 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-util\") pod \"ed012a84-b001-4a5e-b019-cfdf271ef183\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " Sep 18 21:13:34.604772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.604672 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-util\") pod \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " Sep 18 21:13:34.604772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.604695 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58mts\" (UniqueName: \"kubernetes.io/projected/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-kube-api-access-58mts\") pod \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\" (UID: \"7e30e51a-626c-4213-ba4d-9f84ed3b8b48\") " Sep 18 21:13:34.604772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.604723 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-bundle\") pod \"ed012a84-b001-4a5e-b019-cfdf271ef183\" (UID: \"ed012a84-b001-4a5e-b019-cfdf271ef183\") " Sep 18 21:13:34.605267 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.605239 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-bundle" (OuterVolumeSpecName: "bundle") pod "7e30e51a-626c-4213-ba4d-9f84ed3b8b48" (UID: "7e30e51a-626c-4213-ba4d-9f84ed3b8b48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:13:34.605426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.605401 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-bundle" (OuterVolumeSpecName: "bundle") pod "ed012a84-b001-4a5e-b019-cfdf271ef183" (UID: "ed012a84-b001-4a5e-b019-cfdf271ef183"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:13:34.606897 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.606870 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-kube-api-access-58mts" (OuterVolumeSpecName: "kube-api-access-58mts") pod "7e30e51a-626c-4213-ba4d-9f84ed3b8b48" (UID: "7e30e51a-626c-4213-ba4d-9f84ed3b8b48"). InnerVolumeSpecName "kube-api-access-58mts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:13:34.607190 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.607171 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed012a84-b001-4a5e-b019-cfdf271ef183-kube-api-access-sz24g" (OuterVolumeSpecName: "kube-api-access-sz24g") pod "ed012a84-b001-4a5e-b019-cfdf271ef183" (UID: "ed012a84-b001-4a5e-b019-cfdf271ef183"). InnerVolumeSpecName "kube-api-access-sz24g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:13:34.610238 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.610217 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-util" (OuterVolumeSpecName: "util") pod "7e30e51a-626c-4213-ba4d-9f84ed3b8b48" (UID: "7e30e51a-626c-4213-ba4d-9f84ed3b8b48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:13:34.610511 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.610492 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-util" (OuterVolumeSpecName: "util") pod "ed012a84-b001-4a5e-b019-cfdf271ef183" (UID: "ed012a84-b001-4a5e-b019-cfdf271ef183"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:13:34.706191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.706101 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-util\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:34.706191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.706133 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-util\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:34.706191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.706143 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mts\" (UniqueName: \"kubernetes.io/projected/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-kube-api-access-58mts\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:34.706191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.706156 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed012a84-b001-4a5e-b019-cfdf271ef183-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:34.706191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.706165 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz24g\" (UniqueName: \"kubernetes.io/projected/ed012a84-b001-4a5e-b019-cfdf271ef183-kube-api-access-sz24g\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:34.706191 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:34.706173 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e30e51a-626c-4213-ba4d-9f84ed3b8b48-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:35.379684 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.379663 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" Sep 18 21:13:35.380011 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.379682 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dsnhx" event={"ID":"7e30e51a-626c-4213-ba4d-9f84ed3b8b48","Type":"ContainerDied","Data":"683dfc985c1050c59cf0be7a9f5f0d38d7c40cad60e6ff57029553cab91fd5e2"} Sep 18 21:13:35.380011 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.379721 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="683dfc985c1050c59cf0be7a9f5f0d38d7c40cad60e6ff57029553cab91fd5e2" Sep 18 21:13:35.381785 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.381766 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" Sep 18 21:13:35.381785 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.381772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72txbl" event={"ID":"ed012a84-b001-4a5e-b019-cfdf271ef183","Type":"ContainerDied","Data":"8b60ebb786eb926d5bf8e9c5f67084d5ca04652a29bd29ee29bdb918c7d24feb"} Sep 18 21:13:35.381893 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.381819 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b60ebb786eb926d5bf8e9c5f67084d5ca04652a29bd29ee29bdb918c7d24feb" Sep 18 21:13:35.520587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.520566 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:35.532076 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.532054 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:35.614723 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.614692 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-util\") pod \"12ec0ce3-c231-4df7-9abf-0408ff69f947\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " Sep 18 21:13:35.614849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.614805 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-util\") pod \"5e8684e6-c38c-4482-9485-c91770475fa9\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " Sep 18 21:13:35.614849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.614835 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-bundle\") pod \"5e8684e6-c38c-4482-9485-c91770475fa9\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " Sep 18 21:13:35.614931 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.614872 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5hhj\" (UniqueName: \"kubernetes.io/projected/12ec0ce3-c231-4df7-9abf-0408ff69f947-kube-api-access-x5hhj\") pod \"12ec0ce3-c231-4df7-9abf-0408ff69f947\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " Sep 18 21:13:35.614931 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.614911 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbslm\" (UniqueName: \"kubernetes.io/projected/5e8684e6-c38c-4482-9485-c91770475fa9-kube-api-access-gbslm\") pod \"5e8684e6-c38c-4482-9485-c91770475fa9\" (UID: \"5e8684e6-c38c-4482-9485-c91770475fa9\") " Sep 18 21:13:35.615070 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.614965 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-bundle\") pod \"12ec0ce3-c231-4df7-9abf-0408ff69f947\" (UID: \"12ec0ce3-c231-4df7-9abf-0408ff69f947\") " Sep 18 21:13:35.615661 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.615538 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-bundle" (OuterVolumeSpecName: "bundle") pod "5e8684e6-c38c-4482-9485-c91770475fa9" (UID: "5e8684e6-c38c-4482-9485-c91770475fa9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:13:35.616015 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.615977 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-bundle" (OuterVolumeSpecName: "bundle") pod "12ec0ce3-c231-4df7-9abf-0408ff69f947" (UID: "12ec0ce3-c231-4df7-9abf-0408ff69f947"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:13:35.617216 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.617162 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8684e6-c38c-4482-9485-c91770475fa9-kube-api-access-gbslm" (OuterVolumeSpecName: "kube-api-access-gbslm") pod "5e8684e6-c38c-4482-9485-c91770475fa9" (UID: "5e8684e6-c38c-4482-9485-c91770475fa9"). InnerVolumeSpecName "kube-api-access-gbslm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:13:35.617318 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.617302 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ec0ce3-c231-4df7-9abf-0408ff69f947-kube-api-access-x5hhj" (OuterVolumeSpecName: "kube-api-access-x5hhj") pod "12ec0ce3-c231-4df7-9abf-0408ff69f947" (UID: "12ec0ce3-c231-4df7-9abf-0408ff69f947"). InnerVolumeSpecName "kube-api-access-x5hhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:13:35.619778 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.619761 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-util" (OuterVolumeSpecName: "util") pod "12ec0ce3-c231-4df7-9abf-0408ff69f947" (UID: "12ec0ce3-c231-4df7-9abf-0408ff69f947"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:13:35.620942 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.620922 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-util" (OuterVolumeSpecName: "util") pod "5e8684e6-c38c-4482-9485-c91770475fa9" (UID: "5e8684e6-c38c-4482-9485-c91770475fa9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:13:35.716617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.716583 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5hhj\" (UniqueName: \"kubernetes.io/projected/12ec0ce3-c231-4df7-9abf-0408ff69f947-kube-api-access-x5hhj\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:35.716617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.716611 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbslm\" (UniqueName: \"kubernetes.io/projected/5e8684e6-c38c-4482-9485-c91770475fa9-kube-api-access-gbslm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:35.716617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.716621 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:35.716845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.716630 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ec0ce3-c231-4df7-9abf-0408ff69f947-util\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:35.716845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.716638 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-util\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:35.716845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:35.716645 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8684e6-c38c-4482-9485-c91770475fa9-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:36.389183 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:36.389155 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" Sep 18 21:13:36.389602 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:36.389152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tvnq" event={"ID":"12ec0ce3-c231-4df7-9abf-0408ff69f947","Type":"ContainerDied","Data":"3bbf08ee001228d6b5c74fe5ab9a5e7e4b3284b216d6439e627c72755cc86f73"} Sep 18 21:13:36.389602 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:36.389278 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bbf08ee001228d6b5c74fe5ab9a5e7e4b3284b216d6439e627c72755cc86f73" Sep 18 21:13:36.391163 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:36.391147 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" Sep 18 21:13:36.391270 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:36.391143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2q47h" event={"ID":"5e8684e6-c38c-4482-9485-c91770475fa9","Type":"ContainerDied","Data":"a7513f133a9021a28b2ff7c750c3d8b6303bb784b2904dcb4034bc406ab1b55d"} Sep 18 21:13:36.391270 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:36.391261 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7513f133a9021a28b2ff7c750c3d8b6303bb784b2904dcb4034bc406ab1b55d" Sep 18 21:13:44.281279 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281252 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4"] Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281570 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerName="pull" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281581 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerName="pull" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281588 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8684e6-c38c-4482-9485-c91770475fa9" containerName="pull" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281593 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8684e6-c38c-4482-9485-c91770475fa9" containerName="pull" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281599 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerName="extract" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281605 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerName="extract" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281612 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerName="pull" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281617 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerName="pull" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281626 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerName="pull" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281631 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerName="pull" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281642 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8684e6-c38c-4482-9485-c91770475fa9" containerName="util" Sep 18 21:13:44.281642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281647 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8684e6-c38c-4482-9485-c91770475fa9" containerName="util" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281654 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8684e6-c38c-4482-9485-c91770475fa9" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281659 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8684e6-c38c-4482-9485-c91770475fa9" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281669 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerName="util" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281673 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerName="util" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281681 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281696 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281702 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281707 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281714 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerName="util" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281719 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerName="util" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281745 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerName="util" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281750 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerName="util" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281798 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed012a84-b001-4a5e-b019-cfdf271ef183" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281804 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8684e6-c38c-4482-9485-c91770475fa9" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281809 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ec0ce3-c231-4df7-9abf-0408ff69f947" containerName="extract" Sep 18 21:13:44.282032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.281814 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e30e51a-626c-4213-ba4d-9f84ed3b8b48" containerName="extract" Sep 18 21:13:44.283702 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.283685 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:13:44.287474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.287449 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-659jr\"" Sep 18 21:13:44.288124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.288109 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 18 21:13:44.292620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.292587 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 18 21:13:44.303371 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.303348 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4"] Sep 18 21:13:44.389242 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.389213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7ml\" (UniqueName: \"kubernetes.io/projected/9795bef4-6926-40ac-9deb-3961fc1379e1-kube-api-access-mb7ml\") pod \"limitador-operator-controller-manager-74495c69f8-ppcp4\" (UID: \"9795bef4-6926-40ac-9deb-3961fc1379e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:13:44.489938 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.489901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7ml\" (UniqueName: \"kubernetes.io/projected/9795bef4-6926-40ac-9deb-3961fc1379e1-kube-api-access-mb7ml\") pod \"limitador-operator-controller-manager-74495c69f8-ppcp4\" (UID: \"9795bef4-6926-40ac-9deb-3961fc1379e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:13:44.503896 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.503877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7ml\" (UniqueName: \"kubernetes.io/projected/9795bef4-6926-40ac-9deb-3961fc1379e1-kube-api-access-mb7ml\") pod \"limitador-operator-controller-manager-74495c69f8-ppcp4\" (UID: \"9795bef4-6926-40ac-9deb-3961fc1379e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:13:44.593773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.593724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:13:44.873242 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:44.873215 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4"] Sep 18 21:13:44.874875 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:44.874845 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9795bef4_6926_40ac_9deb_3961fc1379e1.slice/crio-4bb6ea2f2fafe974e7e049322182d278c6c41a3e6092f750898df17af9e04b23 WatchSource:0}: Error finding container 4bb6ea2f2fafe974e7e049322182d278c6c41a3e6092f750898df17af9e04b23: Status 404 returned error can't find the container with id 4bb6ea2f2fafe974e7e049322182d278c6c41a3e6092f750898df17af9e04b23 Sep 18 21:13:45.435742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:45.435709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" event={"ID":"9795bef4-6926-40ac-9deb-3961fc1379e1","Type":"ContainerStarted","Data":"4bb6ea2f2fafe974e7e049322182d278c6c41a3e6092f750898df17af9e04b23"} Sep 18 21:13:46.903444 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:46.903414 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dph4s"] Sep 18 21:13:46.906440 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:46.906421 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" Sep 18 21:13:46.909111 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:46.909092 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-w7bvx\"" Sep 18 21:13:46.928761 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:46.928737 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dph4s"] Sep 18 21:13:47.014227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:47.014193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6rd\" (UniqueName: \"kubernetes.io/projected/2c649398-ea3a-4509-b35d-99e6808e00b0-kube-api-access-ml6rd\") pod \"authorino-operator-76b478bd78-dph4s\" (UID: \"2c649398-ea3a-4509-b35d-99e6808e00b0\") " pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" Sep 18 21:13:47.115005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:47.114964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6rd\" (UniqueName: \"kubernetes.io/projected/2c649398-ea3a-4509-b35d-99e6808e00b0-kube-api-access-ml6rd\") pod \"authorino-operator-76b478bd78-dph4s\" (UID: \"2c649398-ea3a-4509-b35d-99e6808e00b0\") " pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" Sep 18 21:13:47.142287 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:47.142252 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6rd\" (UniqueName: \"kubernetes.io/projected/2c649398-ea3a-4509-b35d-99e6808e00b0-kube-api-access-ml6rd\") pod \"authorino-operator-76b478bd78-dph4s\" (UID: \"2c649398-ea3a-4509-b35d-99e6808e00b0\") " pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" Sep 18 21:13:47.218889 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:47.218869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" Sep 18 21:13:47.448848 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:47.448808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" event={"ID":"9795bef4-6926-40ac-9deb-3961fc1379e1","Type":"ContainerStarted","Data":"88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60"} Sep 18 21:13:47.449038 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:47.448876 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:13:47.479861 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:47.479761 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" podStartSLOduration=1.154893807 podStartE2EDuration="3.479746459s" totalImagesPullingTime="2.324852652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:13:47.476401617 +0000 UTC m=+456.406147489" watchObservedRunningTime="2026-09-18 21:13:47.479746459 +0000 UTC m=+456.409492330" Sep 18 21:13:47.506153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:47.506126 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dph4s"] Sep 18 21:13:47.507817 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:47.507787 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c649398_ea3a_4509_b35d_99e6808e00b0.slice/crio-2e8c6f547d09cdddf98398b301fe69945e3d7de064bd8542636ba636eae68d20 WatchSource:0}: Error finding container 2e8c6f547d09cdddf98398b301fe69945e3d7de064bd8542636ba636eae68d20: Status 404 returned error can't find the container with id 2e8c6f547d09cdddf98398b301fe69945e3d7de064bd8542636ba636eae68d20 Sep 18 21:13:48.449521 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:48.449477 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f6f8797c8-w8wmq" podUID="566ad78a-4eb0-4ac0-99ad-c356141898aa" containerName="console" containerID="cri-o://5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50" gracePeriod=15 Sep 18 21:13:48.459061 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:48.458926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" event={"ID":"2c649398-ea3a-4509-b35d-99e6808e00b0","Type":"ContainerStarted","Data":"2e8c6f547d09cdddf98398b301fe69945e3d7de064bd8542636ba636eae68d20"} Sep 18 21:13:48.882411 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:48.882384 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:13:49.030221 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030186 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-config\") pod \"566ad78a-4eb0-4ac0-99ad-c356141898aa\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " Sep 18 21:13:49.030365 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030275 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-oauth-serving-cert\") pod \"566ad78a-4eb0-4ac0-99ad-c356141898aa\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " Sep 18 21:13:49.030459 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030437 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-oauth-config\") pod \"566ad78a-4eb0-4ac0-99ad-c356141898aa\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " Sep 18 21:13:49.030532 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030502 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-trusted-ca-bundle\") pod \"566ad78a-4eb0-4ac0-99ad-c356141898aa\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " Sep 18 21:13:49.030587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030551 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-config" (OuterVolumeSpecName: "console-config") pod "566ad78a-4eb0-4ac0-99ad-c356141898aa" (UID: "566ad78a-4eb0-4ac0-99ad-c356141898aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:13:49.030644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030595 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp5pj\" (UniqueName: \"kubernetes.io/projected/566ad78a-4eb0-4ac0-99ad-c356141898aa-kube-api-access-fp5pj\") pod \"566ad78a-4eb0-4ac0-99ad-c356141898aa\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " Sep 18 21:13:49.030644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030622 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-service-ca\") pod \"566ad78a-4eb0-4ac0-99ad-c356141898aa\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " Sep 18 21:13:49.030644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030623 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "566ad78a-4eb0-4ac0-99ad-c356141898aa" (UID: "566ad78a-4eb0-4ac0-99ad-c356141898aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:13:49.030790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030649 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-serving-cert\") pod \"566ad78a-4eb0-4ac0-99ad-c356141898aa\" (UID: \"566ad78a-4eb0-4ac0-99ad-c356141898aa\") " Sep 18 21:13:49.030920 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030905 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-config\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:49.031002 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030910 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "566ad78a-4eb0-4ac0-99ad-c356141898aa" (UID: "566ad78a-4eb0-4ac0-99ad-c356141898aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:13:49.031002 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030925 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-oauth-serving-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:49.031102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.030996 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "566ad78a-4eb0-4ac0-99ad-c356141898aa" (UID: "566ad78a-4eb0-4ac0-99ad-c356141898aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:13:49.032749 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.032723 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "566ad78a-4eb0-4ac0-99ad-c356141898aa" (UID: "566ad78a-4eb0-4ac0-99ad-c356141898aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:13:49.032839 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.032756 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566ad78a-4eb0-4ac0-99ad-c356141898aa-kube-api-access-fp5pj" (OuterVolumeSpecName: "kube-api-access-fp5pj") pod "566ad78a-4eb0-4ac0-99ad-c356141898aa" (UID: "566ad78a-4eb0-4ac0-99ad-c356141898aa"). InnerVolumeSpecName "kube-api-access-fp5pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:13:49.032883 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.032867 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "566ad78a-4eb0-4ac0-99ad-c356141898aa" (UID: "566ad78a-4eb0-4ac0-99ad-c356141898aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:13:49.131721 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.131649 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-oauth-config\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:49.131721 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.131676 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-trusted-ca-bundle\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:49.131721 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.131685 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp5pj\" (UniqueName: \"kubernetes.io/projected/566ad78a-4eb0-4ac0-99ad-c356141898aa-kube-api-access-fp5pj\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:49.131721 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.131697 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/566ad78a-4eb0-4ac0-99ad-c356141898aa-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:49.131721 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.131706 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/566ad78a-4eb0-4ac0-99ad-c356141898aa-console-serving-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:13:49.466874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.466787 2577 generic.go:356] "Generic (PLEG): container finished" podID="566ad78a-4eb0-4ac0-99ad-c356141898aa" containerID="5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50" exitCode=0 Sep 18 21:13:49.466874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.466830 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f6f8797c8-w8wmq" Sep 18 21:13:49.467389 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.466875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6f8797c8-w8wmq" event={"ID":"566ad78a-4eb0-4ac0-99ad-c356141898aa","Type":"ContainerDied","Data":"5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50"} Sep 18 21:13:49.467389 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.466930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f6f8797c8-w8wmq" event={"ID":"566ad78a-4eb0-4ac0-99ad-c356141898aa","Type":"ContainerDied","Data":"a85f8b40b8a799e813072845924568307fc06b76f99f40d1ccbb37078e9beed8"} Sep 18 21:13:49.467389 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.466988 2577 scope.go:117] "RemoveContainer" containerID="5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50" Sep 18 21:13:49.468704 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.468680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" event={"ID":"2c649398-ea3a-4509-b35d-99e6808e00b0","Type":"ContainerStarted","Data":"0e884659defd7c265be33d5d751c4ae78ff8b1b9004637d671666d3568878e7a"} Sep 18 21:13:49.468842 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.468823 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" Sep 18 21:13:49.475745 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.475718 2577 scope.go:117] "RemoveContainer" containerID="5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50" Sep 18 21:13:49.476007 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:13:49.475986 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50\": container with ID starting with 5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50 not found: ID does not exist" containerID="5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50" Sep 18 21:13:49.476088 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.476018 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50"} err="failed to get container status \"5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50\": rpc error: code = NotFound desc = could not find container \"5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50\": container with ID starting with 5246c80f287d95101ac28d7e8272a676b07f929b1c2114241e8868bceff7ea50 not found: ID does not exist" Sep 18 21:13:49.529464 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.529428 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f6f8797c8-w8wmq"] Sep 18 21:13:49.534716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.534685 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f6f8797c8-w8wmq"] Sep 18 21:13:49.584918 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.584840 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" podStartSLOduration=2.192353653 podStartE2EDuration="3.584823s" totalImagesPullingTime="1.392469347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:13:49.583817787 +0000 UTC m=+458.513563659" watchObservedRunningTime="2026-09-18 21:13:49.584823 +0000 UTC m=+458.514568872" Sep 18 21:13:49.599584 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.599550 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw"] Sep 18 21:13:49.599904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.599892 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566ad78a-4eb0-4ac0-99ad-c356141898aa" containerName="console" Sep 18 21:13:49.599944 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.599905 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="566ad78a-4eb0-4ac0-99ad-c356141898aa" containerName="console" Sep 18 21:13:49.600013 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.599981 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="566ad78a-4eb0-4ac0-99ad-c356141898aa" containerName="console" Sep 18 21:13:49.602613 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.602598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" Sep 18 21:13:49.605922 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.605902 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 18 21:13:49.606041 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.605907 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cf5vx\"" Sep 18 21:13:49.632994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.632941 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw"] Sep 18 21:13:49.737568 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.737486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pdw\" (UniqueName: \"kubernetes.io/projected/5a2f719b-b21e-45a0-9717-b14ef8628bdb-kube-api-access-p7pdw\") pod \"dns-operator-controller-manager-7b84cd7c9c-t26kw\" (UID: \"5a2f719b-b21e-45a0-9717-b14ef8628bdb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" Sep 18 21:13:49.839004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.838950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pdw\" (UniqueName: \"kubernetes.io/projected/5a2f719b-b21e-45a0-9717-b14ef8628bdb-kube-api-access-p7pdw\") pod \"dns-operator-controller-manager-7b84cd7c9c-t26kw\" (UID: \"5a2f719b-b21e-45a0-9717-b14ef8628bdb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" Sep 18 21:13:49.854654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.854627 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pdw\" (UniqueName: \"kubernetes.io/projected/5a2f719b-b21e-45a0-9717-b14ef8628bdb-kube-api-access-p7pdw\") pod \"dns-operator-controller-manager-7b84cd7c9c-t26kw\" (UID: \"5a2f719b-b21e-45a0-9717-b14ef8628bdb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" Sep 18 21:13:49.912627 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:49.912589 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" Sep 18 21:13:50.198051 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:50.198017 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw"] Sep 18 21:13:50.199430 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:50.199405 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2f719b_b21e_45a0_9717_b14ef8628bdb.slice/crio-5caf9ac6b02d2bcc437eacf59ce510eed7448a2bfdb50f601dd0e6d7c9cb518d WatchSource:0}: Error finding container 5caf9ac6b02d2bcc437eacf59ce510eed7448a2bfdb50f601dd0e6d7c9cb518d: Status 404 returned error can't find the container with id 5caf9ac6b02d2bcc437eacf59ce510eed7448a2bfdb50f601dd0e6d7c9cb518d Sep 18 21:13:50.475619 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:50.475538 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" event={"ID":"5a2f719b-b21e-45a0-9717-b14ef8628bdb","Type":"ContainerStarted","Data":"5caf9ac6b02d2bcc437eacf59ce510eed7448a2bfdb50f601dd0e6d7c9cb518d"} Sep 18 21:13:51.359473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:51.359432 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566ad78a-4eb0-4ac0-99ad-c356141898aa" path="/var/lib/kubelet/pods/566ad78a-4eb0-4ac0-99ad-c356141898aa/volumes" Sep 18 21:13:52.489550 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:52.489515 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" event={"ID":"5a2f719b-b21e-45a0-9717-b14ef8628bdb","Type":"ContainerStarted","Data":"dce8612bd51fe6ea3595dac43af6674eb47b45eddea58de363e2d9bad9c0cf92"} Sep 18 21:13:52.489982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:52.489655 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" Sep 18 21:13:52.518657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:52.518608 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" podStartSLOduration=1.938482993 podStartE2EDuration="3.518593542s" totalImagesPullingTime="1.580110549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:13:52.517520286 +0000 UTC m=+461.447266157" watchObservedRunningTime="2026-09-18 21:13:52.518593542 +0000 UTC m=+461.448339415" Sep 18 21:13:53.925722 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:53.925681 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj"] Sep 18 21:13:53.928100 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:53.928085 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:53.933118 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:53.933085 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zlvvm\"" Sep 18 21:13:53.957812 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:53.957784 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj"] Sep 18 21:13:54.078946 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:54.078893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2736e941-ef7a-4099-83a3-abbd5a93983b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" (UID: \"2736e941-ef7a-4099-83a3-abbd5a93983b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:54.079154 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:54.079046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkb2w\" (UniqueName: \"kubernetes.io/projected/2736e941-ef7a-4099-83a3-abbd5a93983b-kube-api-access-fkb2w\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" (UID: \"2736e941-ef7a-4099-83a3-abbd5a93983b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:54.180477 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:54.180390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2736e941-ef7a-4099-83a3-abbd5a93983b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" (UID: \"2736e941-ef7a-4099-83a3-abbd5a93983b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:54.180477 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:54.180472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkb2w\" (UniqueName: \"kubernetes.io/projected/2736e941-ef7a-4099-83a3-abbd5a93983b-kube-api-access-fkb2w\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" (UID: \"2736e941-ef7a-4099-83a3-abbd5a93983b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:54.180777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:54.180757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2736e941-ef7a-4099-83a3-abbd5a93983b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" (UID: \"2736e941-ef7a-4099-83a3-abbd5a93983b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:54.190696 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:54.190667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkb2w\" (UniqueName: \"kubernetes.io/projected/2736e941-ef7a-4099-83a3-abbd5a93983b-kube-api-access-fkb2w\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" (UID: \"2736e941-ef7a-4099-83a3-abbd5a93983b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:54.237390 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:54.237359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:54.521984 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:54.521894 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj"] Sep 18 21:13:54.525364 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:13:54.525335 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2736e941_ef7a_4099_83a3_abbd5a93983b.slice/crio-60a0fd2dde360a02e63c19b54ea3c6cdbf2ed553d8c1991269a8d2fea2e0c2d1 WatchSource:0}: Error finding container 60a0fd2dde360a02e63c19b54ea3c6cdbf2ed553d8c1991269a8d2fea2e0c2d1: Status 404 returned error can't find the container with id 60a0fd2dde360a02e63c19b54ea3c6cdbf2ed553d8c1991269a8d2fea2e0c2d1 Sep 18 21:13:55.506756 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:55.506721 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" event={"ID":"2736e941-ef7a-4099-83a3-abbd5a93983b","Type":"ContainerStarted","Data":"60a0fd2dde360a02e63c19b54ea3c6cdbf2ed553d8c1991269a8d2fea2e0c2d1"} Sep 18 21:13:58.461997 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:58.461949 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:13:59.535196 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:59.535162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" event={"ID":"2736e941-ef7a-4099-83a3-abbd5a93983b","Type":"ContainerStarted","Data":"ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584"} Sep 18 21:13:59.535586 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:59.535240 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:13:59.559631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:13:59.559581 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" podStartSLOduration=2.547013053 podStartE2EDuration="6.559564496s" totalImagesPullingTime="4.012551443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:13:59.558151318 +0000 UTC m=+468.487897192" watchObservedRunningTime="2026-09-18 21:13:59.559564496 +0000 UTC m=+468.489310368" Sep 18 21:14:00.479296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:00.479264 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-dph4s" Sep 18 21:14:01.373924 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.373891 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f"] Sep 18 21:14:01.377673 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.377656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.382993 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.382943 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 18 21:14:01.383370 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.383350 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 18 21:14:01.383585 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.383567 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sj6pb\"" Sep 18 21:14:01.396557 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.396539 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f"] Sep 18 21:14:01.550645 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.550617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.550849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.550655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xl5w\" (UniqueName: \"kubernetes.io/projected/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-kube-api-access-6xl5w\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.550849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.550684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.651371 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.651290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xl5w\" (UniqueName: \"kubernetes.io/projected/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-kube-api-access-6xl5w\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.651371 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.651341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.651571 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.651439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.652085 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.652059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.653765 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.653745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.675406 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.675380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xl5w\" (UniqueName: \"kubernetes.io/projected/2e6e665b-9f29-4cfc-8dde-80c9639d2fda-kube-api-access-6xl5w\") pod \"kuadrant-console-plugin-fbdd4f445-g956f\" (UID: \"2e6e665b-9f29-4cfc-8dde-80c9639d2fda\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.687236 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.687206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" Sep 18 21:14:01.962492 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:01.962465 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f"] Sep 18 21:14:01.965008 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:14:01.964977 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e6e665b_9f29_4cfc_8dde_80c9639d2fda.slice/crio-91f72a46dfeba32d0dff5560a86bf7b7dd36e8fa8480250bef7c1ae4b387742f WatchSource:0}: Error finding container 91f72a46dfeba32d0dff5560a86bf7b7dd36e8fa8480250bef7c1ae4b387742f: Status 404 returned error can't find the container with id 91f72a46dfeba32d0dff5560a86bf7b7dd36e8fa8480250bef7c1ae4b387742f Sep 18 21:14:02.553602 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:02.553568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" event={"ID":"2e6e665b-9f29-4cfc-8dde-80c9639d2fda","Type":"ContainerStarted","Data":"91f72a46dfeba32d0dff5560a86bf7b7dd36e8fa8480250bef7c1ae4b387742f"} Sep 18 21:14:03.498236 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:03.498206 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-t26kw" Sep 18 21:14:06.582068 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:06.582041 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" event={"ID":"2e6e665b-9f29-4cfc-8dde-80c9639d2fda","Type":"ContainerStarted","Data":"b6c8719bf49f7f3b708b9d01d71fccc46dbd141a268c3d8f4148ab40edfe3ef2"} Sep 18 21:14:06.601050 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:06.601009 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-g956f" podStartSLOduration=1.052750843 podStartE2EDuration="5.600995214s" totalImagesPullingTime="4.548244371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:06.59962224 +0000 UTC m=+475.529368114" watchObservedRunningTime="2026-09-18 21:14:06.600995214 +0000 UTC m=+475.530741088" Sep 18 21:14:10.544994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:10.544935 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:12.886984 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:12.886936 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj"] Sep 18 21:14:12.887381 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:12.887137 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" containerName="manager" containerID="cri-o://ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584" gracePeriod=2 Sep 18 21:14:12.939077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:12.939049 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj"] Sep 18 21:14:12.972591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:12.972557 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4"] Sep 18 21:14:12.972828 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:12.972804 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" containerName="manager" containerID="cri-o://88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60" gracePeriod=2 Sep 18 21:14:12.989506 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:12.989482 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4"] Sep 18 21:14:13.011007 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.010974 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:13.049277 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.049244 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2"] Sep 18 21:14:13.049819 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.049803 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" containerName="manager" Sep 18 21:14:13.049892 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.049822 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" containerName="manager" Sep 18 21:14:13.049892 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.049842 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" containerName="manager" Sep 18 21:14:13.049892 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.049850 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" containerName="manager" Sep 18 21:14:13.050045 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.049927 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" containerName="manager" Sep 18 21:14:13.050045 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.049943 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" containerName="manager" Sep 18 21:14:13.053368 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.053349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:13.072014 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.071989 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9"] Sep 18 21:14:13.075335 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.075312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" Sep 18 21:14:13.091519 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.091492 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2"] Sep 18 21:14:13.112449 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.112371 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9"] Sep 18 21:14:13.153718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.153500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bnp\" (UniqueName: \"kubernetes.io/projected/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-kube-api-access-67bnp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w2x2\" (UID: \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:13.153718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.153565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w2x2\" (UID: \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:13.153718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.153608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4br\" (UniqueName: \"kubernetes.io/projected/9bc9fac1-f157-4505-85de-16b78d21a1fe-kube-api-access-sv4br\") pod \"limitador-operator-controller-manager-74495c69f8-gc4g9\" (UID: \"9bc9fac1-f157-4505-85de-16b78d21a1fe\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" Sep 18 21:14:13.157624 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.157588 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:13.161791 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.160191 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ppcp4\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:14:13.164984 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.162689 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:13.169537 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.169506 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ppcp4\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:14:13.228425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.228392 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:13.232102 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.232062 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:13.234426 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.234394 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ppcp4\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:14:13.236450 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.236435 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:14:13.240798 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.240777 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:13.244137 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.244119 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ppcp4\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:14:13.254480 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.254464 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkb2w\" (UniqueName: \"kubernetes.io/projected/2736e941-ef7a-4099-83a3-abbd5a93983b-kube-api-access-fkb2w\") pod \"2736e941-ef7a-4099-83a3-abbd5a93983b\" (UID: \"2736e941-ef7a-4099-83a3-abbd5a93983b\") " Sep 18 21:14:13.254547 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.254494 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2736e941-ef7a-4099-83a3-abbd5a93983b-extensions-socket-volume\") pod \"2736e941-ef7a-4099-83a3-abbd5a93983b\" (UID: \"2736e941-ef7a-4099-83a3-abbd5a93983b\") " Sep 18 21:14:13.254547 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.254526 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb7ml\" (UniqueName: \"kubernetes.io/projected/9795bef4-6926-40ac-9deb-3961fc1379e1-kube-api-access-mb7ml\") pod \"9795bef4-6926-40ac-9deb-3961fc1379e1\" (UID: \"9795bef4-6926-40ac-9deb-3961fc1379e1\") " Sep 18 21:14:13.254717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.254696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4br\" (UniqueName: \"kubernetes.io/projected/9bc9fac1-f157-4505-85de-16b78d21a1fe-kube-api-access-sv4br\") pod \"limitador-operator-controller-manager-74495c69f8-gc4g9\" (UID: \"9bc9fac1-f157-4505-85de-16b78d21a1fe\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" Sep 18 21:14:13.254849 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.254832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bnp\" (UniqueName: \"kubernetes.io/projected/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-kube-api-access-67bnp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w2x2\" (UID: \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:13.254913 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.254882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w2x2\" (UID: \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:13.255022 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.254982 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2736e941-ef7a-4099-83a3-abbd5a93983b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2736e941-ef7a-4099-83a3-abbd5a93983b" (UID: "2736e941-ef7a-4099-83a3-abbd5a93983b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:14:13.255192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.255177 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w2x2\" (UID: \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:13.257229 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.257208 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9795bef4-6926-40ac-9deb-3961fc1379e1-kube-api-access-mb7ml" (OuterVolumeSpecName: "kube-api-access-mb7ml") pod "9795bef4-6926-40ac-9deb-3961fc1379e1" (UID: "9795bef4-6926-40ac-9deb-3961fc1379e1"). InnerVolumeSpecName "kube-api-access-mb7ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:14:13.257229 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.257217 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2736e941-ef7a-4099-83a3-abbd5a93983b-kube-api-access-fkb2w" (OuterVolumeSpecName: "kube-api-access-fkb2w") pod "2736e941-ef7a-4099-83a3-abbd5a93983b" (UID: "2736e941-ef7a-4099-83a3-abbd5a93983b"). InnerVolumeSpecName "kube-api-access-fkb2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:14:13.268064 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.268044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4br\" (UniqueName: \"kubernetes.io/projected/9bc9fac1-f157-4505-85de-16b78d21a1fe-kube-api-access-sv4br\") pod \"limitador-operator-controller-manager-74495c69f8-gc4g9\" (UID: \"9bc9fac1-f157-4505-85de-16b78d21a1fe\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" Sep 18 21:14:13.272463 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.272445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bnp\" (UniqueName: \"kubernetes.io/projected/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-kube-api-access-67bnp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w2x2\" (UID: \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:13.355614 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.355580 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkb2w\" (UniqueName: \"kubernetes.io/projected/2736e941-ef7a-4099-83a3-abbd5a93983b-kube-api-access-fkb2w\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:14:13.355614 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.355610 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2736e941-ef7a-4099-83a3-abbd5a93983b-extensions-socket-volume\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:14:13.355799 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.355625 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7ml\" (UniqueName: \"kubernetes.io/projected/9795bef4-6926-40ac-9deb-3961fc1379e1-kube-api-access-mb7ml\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:14:13.358007 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.357979 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-5rllj\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:13.358569 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.358549 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2736e941-ef7a-4099-83a3-abbd5a93983b" path="/var/lib/kubelet/pods/2736e941-ef7a-4099-83a3-abbd5a93983b/volumes" Sep 18 21:14:13.358864 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.358852 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" path="/var/lib/kubelet/pods/9795bef4-6926-40ac-9deb-3961fc1379e1/volumes" Sep 18 21:14:13.360342 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.360320 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ppcp4\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="9795bef4-6926-40ac-9deb-3961fc1379e1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:14:13.366907 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.366888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:13.386867 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.386844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" Sep 18 21:14:13.624452 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.624417 2577 generic.go:356] "Generic (PLEG): container finished" podID="2736e941-ef7a-4099-83a3-abbd5a93983b" containerID="ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584" exitCode=0 Sep 18 21:14:13.624660 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.624471 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5rllj" Sep 18 21:14:13.624660 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.624532 2577 scope.go:117] "RemoveContainer" containerID="ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584" Sep 18 21:14:13.626248 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.626223 2577 generic.go:356] "Generic (PLEG): container finished" podID="9795bef4-6926-40ac-9deb-3961fc1379e1" containerID="88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60" exitCode=0 Sep 18 21:14:13.626425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.626273 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ppcp4" Sep 18 21:14:13.634376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.634356 2577 scope.go:117] "RemoveContainer" containerID="ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584" Sep 18 21:14:13.634650 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.634632 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584\": container with ID starting with ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584 not found: ID does not exist" containerID="ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584" Sep 18 21:14:13.634715 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.634655 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584"} err="failed to get container status \"ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584\": rpc error: code = NotFound desc = could not find container \"ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584\": container with ID starting with ccd591c33532105b11f093384b5091ff5b008685082bccfa841f922e36399584 not found: ID does not exist" Sep 18 21:14:13.634715 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.634672 2577 scope.go:117] "RemoveContainer" containerID="88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60" Sep 18 21:14:13.642209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.642193 2577 scope.go:117] "RemoveContainer" containerID="88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60" Sep 18 21:14:13.642489 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:13.642471 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60\": container with ID starting with 88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60 not found: ID does not exist" containerID="88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60" Sep 18 21:14:13.642536 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.642495 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60"} err="failed to get container status \"88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60\": rpc error: code = NotFound desc = could not find container \"88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60\": container with ID starting with 88b9438433fc6f44ce271c31ce310eebce02883d7e0f3ecc0bfcb7bda7677b60 not found: ID does not exist" Sep 18 21:14:13.681907 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.681780 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2"] Sep 18 21:14:13.684239 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:14:13.684215 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2a5765_3e1f_41fd_b901_f4f3e9e5b92b.slice/crio-0975ac0f67a859602ce942e26bea7ef4f85f1e33d9cd57201ea5e7130f9d280c WatchSource:0}: Error finding container 0975ac0f67a859602ce942e26bea7ef4f85f1e33d9cd57201ea5e7130f9d280c: Status 404 returned error can't find the container with id 0975ac0f67a859602ce942e26bea7ef4f85f1e33d9cd57201ea5e7130f9d280c Sep 18 21:14:13.694595 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:13.694573 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9"] Sep 18 21:14:13.697674 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:14:13.697651 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc9fac1_f157_4505_85de_16b78d21a1fe.slice/crio-9dcc3b39669c82b30d6e555d5287e14c751a91e582c45c8a51ab3a417435ec4d WatchSource:0}: Error finding container 9dcc3b39669c82b30d6e555d5287e14c751a91e582c45c8a51ab3a417435ec4d: Status 404 returned error can't find the container with id 9dcc3b39669c82b30d6e555d5287e14c751a91e582c45c8a51ab3a417435ec4d Sep 18 21:14:14.635547 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:14.635505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" event={"ID":"9bc9fac1-f157-4505-85de-16b78d21a1fe","Type":"ContainerStarted","Data":"b831219a70323fa0edc8a721b14b54c0220fb695c2125f40dec0e5b6aa0e3bcf"} Sep 18 21:14:14.636077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:14.635569 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" event={"ID":"9bc9fac1-f157-4505-85de-16b78d21a1fe","Type":"ContainerStarted","Data":"9dcc3b39669c82b30d6e555d5287e14c751a91e582c45c8a51ab3a417435ec4d"} Sep 18 21:14:14.636077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:14.635596 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" Sep 18 21:14:14.637313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:14.637290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" event={"ID":"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b","Type":"ContainerStarted","Data":"f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a"} Sep 18 21:14:14.637432 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:14.637331 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" event={"ID":"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b","Type":"ContainerStarted","Data":"0975ac0f67a859602ce942e26bea7ef4f85f1e33d9cd57201ea5e7130f9d280c"} Sep 18 21:14:14.637432 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:14.637394 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:14.684201 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:14.684154 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" podStartSLOduration=1.6841384640000001 podStartE2EDuration="1.684138464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:14.681673725 +0000 UTC m=+483.611419597" watchObservedRunningTime="2026-09-18 21:14:14.684138464 +0000 UTC m=+483.613884338" Sep 18 21:14:14.744434 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:14.744379 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" podStartSLOduration=2.744362981 podStartE2EDuration="2.744362981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:14.742426699 +0000 UTC m=+483.672172572" watchObservedRunningTime="2026-09-18 21:14:14.744362981 +0000 UTC m=+483.674108854" Sep 18 21:14:25.646480 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:25.646448 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gc4g9" Sep 18 21:14:25.647014 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:25.646518 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:28.357003 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.356938 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2"] Sep 18 21:14:28.357473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.357137 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" podUID="4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b" containerName="manager" containerID="cri-o://f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a" gracePeriod=10 Sep 18 21:14:28.614151 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.614091 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:28.684822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.684783 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk"] Sep 18 21:14:28.685255 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.685240 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b" containerName="manager" Sep 18 21:14:28.685314 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.685257 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b" containerName="manager" Sep 18 21:14:28.685350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.685314 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b" containerName="manager" Sep 18 21:14:28.689047 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.689028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:28.694260 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.694238 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67bnp\" (UniqueName: \"kubernetes.io/projected/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-kube-api-access-67bnp\") pod \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\" (UID: \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\") " Sep 18 21:14:28.694386 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.694290 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-extensions-socket-volume\") pod \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\" (UID: \"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b\") " Sep 18 21:14:28.694658 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.694635 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b" (UID: "4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:14:28.696527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.696506 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-kube-api-access-67bnp" (OuterVolumeSpecName: "kube-api-access-67bnp") pod "4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b" (UID: "4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b"). InnerVolumeSpecName "kube-api-access-67bnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:14:28.717553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.717521 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk"] Sep 18 21:14:28.725852 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.725822 2577 generic.go:356] "Generic (PLEG): container finished" podID="4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b" containerID="f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a" exitCode=0 Sep 18 21:14:28.725975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.725871 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" Sep 18 21:14:28.725975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.725891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" event={"ID":"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b","Type":"ContainerDied","Data":"f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a"} Sep 18 21:14:28.725975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.725940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2" event={"ID":"4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b","Type":"ContainerDied","Data":"0975ac0f67a859602ce942e26bea7ef4f85f1e33d9cd57201ea5e7130f9d280c"} Sep 18 21:14:28.726101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.725982 2577 scope.go:117] "RemoveContainer" containerID="f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a" Sep 18 21:14:28.735547 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.735527 2577 scope.go:117] "RemoveContainer" containerID="f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a" Sep 18 21:14:28.735807 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:14:28.735790 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a\": container with ID starting with f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a not found: ID does not exist" containerID="f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a" Sep 18 21:14:28.735878 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.735816 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a"} err="failed to get container status \"f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a\": rpc error: code = NotFound desc = could not find container \"f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a\": container with ID starting with f7b713eb6f73833fa50ce93a4fac8e212287b306afef6539c2e40a0c3954246a not found: ID does not exist" Sep 18 21:14:28.777082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.777057 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2"] Sep 18 21:14:28.783056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.783036 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w2x2"] Sep 18 21:14:28.795563 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.795545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aefa99f6-20fd-4967-a7a6-dd44d9969866-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jrzk\" (UID: \"aefa99f6-20fd-4967-a7a6-dd44d9969866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:28.795633 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.795592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjnmj\" (UniqueName: \"kubernetes.io/projected/aefa99f6-20fd-4967-a7a6-dd44d9969866-kube-api-access-pjnmj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jrzk\" (UID: \"aefa99f6-20fd-4967-a7a6-dd44d9969866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:28.795682 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.795658 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67bnp\" (UniqueName: \"kubernetes.io/projected/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-kube-api-access-67bnp\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:14:28.795682 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.795668 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b-extensions-socket-volume\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:14:28.896728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.896657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aefa99f6-20fd-4967-a7a6-dd44d9969866-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jrzk\" (UID: \"aefa99f6-20fd-4967-a7a6-dd44d9969866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:28.896728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.896708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjnmj\" (UniqueName: \"kubernetes.io/projected/aefa99f6-20fd-4967-a7a6-dd44d9969866-kube-api-access-pjnmj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jrzk\" (UID: \"aefa99f6-20fd-4967-a7a6-dd44d9969866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:28.897051 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.897034 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aefa99f6-20fd-4967-a7a6-dd44d9969866-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jrzk\" (UID: \"aefa99f6-20fd-4967-a7a6-dd44d9969866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:28.908883 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.908862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjnmj\" (UniqueName: \"kubernetes.io/projected/aefa99f6-20fd-4967-a7a6-dd44d9969866-kube-api-access-pjnmj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jrzk\" (UID: \"aefa99f6-20fd-4967-a7a6-dd44d9969866\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:28.998692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:28.998658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:29.286611 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:14:29.286578 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefa99f6_20fd_4967_a7a6_dd44d9969866.slice/crio-63a61385964d27cf3d2b0f5ca28b0985ab9ea0afaa256c7f0324796416738a28 WatchSource:0}: Error finding container 63a61385964d27cf3d2b0f5ca28b0985ab9ea0afaa256c7f0324796416738a28: Status 404 returned error can't find the container with id 63a61385964d27cf3d2b0f5ca28b0985ab9ea0afaa256c7f0324796416738a28 Sep 18 21:14:29.286790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:29.286767 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk"] Sep 18 21:14:29.359409 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:29.359379 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b" path="/var/lib/kubelet/pods/4d2a5765-3e1f-41fd-b901-f4f3e9e5b92b/volumes" Sep 18 21:14:29.735725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:29.735693 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" event={"ID":"aefa99f6-20fd-4967-a7a6-dd44d9969866","Type":"ContainerStarted","Data":"8beaaf1dbb0dc2cd386652b4a181b3478153ac67d3e0ebde506e1b228912d820"} Sep 18 21:14:29.735919 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:29.735748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" event={"ID":"aefa99f6-20fd-4967-a7a6-dd44d9969866","Type":"ContainerStarted","Data":"63a61385964d27cf3d2b0f5ca28b0985ab9ea0afaa256c7f0324796416738a28"} Sep 18 21:14:29.735919 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:29.735856 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:40.745668 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:40.745633 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:14:40.774583 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:40.774531 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" podStartSLOduration=12.774515059 podStartE2EDuration="12.774515059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:29.780259255 +0000 UTC m=+498.710005128" watchObservedRunningTime="2026-09-18 21:14:40.774515059 +0000 UTC m=+509.704260931" Sep 18 21:14:44.889016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:44.888981 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x"] Sep 18 21:14:44.906399 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:44.906368 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:44.908888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:44.908861 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kp8sx\"" Sep 18 21:14:44.919238 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:44.919211 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x"] Sep 18 21:14:45.038418 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038378 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.038418 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4zd\" (UniqueName: \"kubernetes.io/projected/0da2b319-54e0-4adc-928c-cf7faf5849be-kube-api-access-zw4zd\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.038641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.038641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.038641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da2b319-54e0-4adc-928c-cf7faf5849be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.038641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.038641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.038802 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.038802 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.038670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.139645 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.139814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.139814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4zd\" (UniqueName: \"kubernetes.io/projected/0da2b319-54e0-4adc-928c-cf7faf5849be-kube-api-access-zw4zd\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.139814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.139814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.139814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da2b319-54e0-4adc-928c-cf7faf5849be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.139814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.139814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.140207 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.139846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.140207 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.140077 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.140207 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.140130 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.140361 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.140306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.140409 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.140372 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.140491 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.140473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da2b319-54e0-4adc-928c-cf7faf5849be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.142148 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.142116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.142285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.142265 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.158446 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.158406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da2b319-54e0-4adc-928c-cf7faf5849be-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.158591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.158514 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4zd\" (UniqueName: \"kubernetes.io/projected/0da2b319-54e0-4adc-928c-cf7faf5849be-kube-api-access-zw4zd\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j9d9x\" (UID: \"0da2b319-54e0-4adc-928c-cf7faf5849be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.218285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.218247 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:45.536455 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.536430 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x"] Sep 18 21:14:45.538118 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:14:45.538083 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da2b319_54e0_4adc_928c_cf7faf5849be.slice/crio-f972a8047eb0ca808af2923dba75e51c3fe96699bf3658038c98455e2caca6ad WatchSource:0}: Error finding container f972a8047eb0ca808af2923dba75e51c3fe96699bf3658038c98455e2caca6ad: Status 404 returned error can't find the container with id f972a8047eb0ca808af2923dba75e51c3fe96699bf3658038c98455e2caca6ad Sep 18 21:14:45.540210 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.540179 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:14:45.540292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.540249 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:14:45.540292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.540278 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:14:45.829397 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.829364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" event={"ID":"0da2b319-54e0-4adc-928c-cf7faf5849be","Type":"ContainerStarted","Data":"aeb374f3175747b50d91d9dd2feb37f8771add6bde0afd0bd204b3a154d1ac87"} Sep 18 21:14:45.829636 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.829429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" event={"ID":"0da2b319-54e0-4adc-928c-cf7faf5849be","Type":"ContainerStarted","Data":"f972a8047eb0ca808af2923dba75e51c3fe96699bf3658038c98455e2caca6ad"} Sep 18 21:14:45.857540 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:45.857467 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" podStartSLOduration=1.857447493 podStartE2EDuration="1.857447493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:45.855768583 +0000 UTC m=+514.785514455" watchObservedRunningTime="2026-09-18 21:14:45.857447493 +0000 UTC m=+514.787193366" Sep 18 21:14:46.218536 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:46.218436 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:46.219738 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:46.219714 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j9d9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 18 21:14:46.219811 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:46.219764 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" podUID="0da2b319-54e0-4adc-928c-cf7faf5849be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 18 21:14:47.223407 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:47.223378 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:47.842430 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:47.842391 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:47.843321 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:47.843301 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:14:57.578150 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:57.578066 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2cxsk"] Sep 18 21:14:57.582586 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:57.582549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2cxsk" Sep 18 21:14:57.587174 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:57.585630 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-95fsz\"" Sep 18 21:14:57.597720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:57.597682 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2cxsk"] Sep 18 21:14:57.757926 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:57.757889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgtk4\" (UniqueName: \"kubernetes.io/projected/0a455b9e-cc1d-4027-b6ef-e18052e575fb-kube-api-access-lgtk4\") pod \"authorino-59c974bd48-2cxsk\" (UID: \"0a455b9e-cc1d-4027-b6ef-e18052e575fb\") " pod="kuadrant-system/authorino-59c974bd48-2cxsk" Sep 18 21:14:57.859279 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:57.859178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgtk4\" (UniqueName: \"kubernetes.io/projected/0a455b9e-cc1d-4027-b6ef-e18052e575fb-kube-api-access-lgtk4\") pod \"authorino-59c974bd48-2cxsk\" (UID: \"0a455b9e-cc1d-4027-b6ef-e18052e575fb\") " pod="kuadrant-system/authorino-59c974bd48-2cxsk" Sep 18 21:14:57.868719 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:57.868683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgtk4\" (UniqueName: \"kubernetes.io/projected/0a455b9e-cc1d-4027-b6ef-e18052e575fb-kube-api-access-lgtk4\") pod \"authorino-59c974bd48-2cxsk\" (UID: \"0a455b9e-cc1d-4027-b6ef-e18052e575fb\") " pod="kuadrant-system/authorino-59c974bd48-2cxsk" Sep 18 21:14:57.894472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:57.894432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2cxsk" Sep 18 21:14:58.168092 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:58.168068 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2cxsk"] Sep 18 21:14:58.169986 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:14:58.169945 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a455b9e_cc1d_4027_b6ef_e18052e575fb.slice/crio-680778256d6a83e679a30a446e2483241dea6863bfd05828e7215c73d00b8cc1 WatchSource:0}: Error finding container 680778256d6a83e679a30a446e2483241dea6863bfd05828e7215c73d00b8cc1: Status 404 returned error can't find the container with id 680778256d6a83e679a30a446e2483241dea6863bfd05828e7215c73d00b8cc1 Sep 18 21:14:58.910188 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:58.910154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2cxsk" event={"ID":"0a455b9e-cc1d-4027-b6ef-e18052e575fb","Type":"ContainerStarted","Data":"680778256d6a83e679a30a446e2483241dea6863bfd05828e7215c73d00b8cc1"} Sep 18 21:14:59.918637 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:14:59.918600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2cxsk" event={"ID":"0a455b9e-cc1d-4027-b6ef-e18052e575fb","Type":"ContainerStarted","Data":"5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4"} Sep 18 21:15:26.761212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.761157 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2cxsk" podStartSLOduration=28.210200318 podStartE2EDuration="29.761139745s" totalImagesPullingTime="1.550939427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:59.955217563 +0000 UTC m=+528.884963438" watchObservedRunningTime="2026-09-18 21:15:26.761139745 +0000 UTC m=+555.690885618" Sep 18 21:15:26.761887 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.761867 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c44d9f6cf-6zs5s"] Sep 18 21:15:26.765357 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.765339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:26.773079 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:15:26.773048 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-6c44d9f6cf-6zs5s\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="d579e5cb-b61b-424e-9869-63d8f45ed068" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:26.773174 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:15:26.773131 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"authorino-server-cert\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" Sep 18 21:15:26.785353 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.785324 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c44d9f6cf-6zs5s"] Sep 18 21:15:26.804119 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.804089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert\") pod \"authorino-6c44d9f6cf-6zs5s\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:26.804272 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.804147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpjg\" (UniqueName: \"kubernetes.io/projected/d579e5cb-b61b-424e-9869-63d8f45ed068-kube-api-access-qhpjg\") pod \"authorino-6c44d9f6cf-6zs5s\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:26.905588 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.905549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert\") pod \"authorino-6c44d9f6cf-6zs5s\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:26.905774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.905610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpjg\" (UniqueName: \"kubernetes.io/projected/d579e5cb-b61b-424e-9869-63d8f45ed068-kube-api-access-qhpjg\") pod \"authorino-6c44d9f6cf-6zs5s\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:26.923462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:26.923432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpjg\" (UniqueName: \"kubernetes.io/projected/d579e5cb-b61b-424e-9869-63d8f45ed068-kube-api-access-qhpjg\") pod \"authorino-6c44d9f6cf-6zs5s\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:27.906601 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:15:27.906562 2577 secret.go:189] Couldn't get secret kuadrant-system/authorino-server-cert: failed to sync secret cache: timed out waiting for the condition Sep 18 21:15:27.907135 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:15:27.906665 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert podName:d579e5cb-b61b-424e-9869-63d8f45ed068 nodeName:}" failed. No retries permitted until 2026-09-18 21:15:28.406644133 +0000 UTC m=+557.336389984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert") pod "authorino-6c44d9f6cf-6zs5s" (UID: "d579e5cb-b61b-424e-9869-63d8f45ed068") : failed to sync secret cache: timed out waiting for the condition Sep 18 21:15:28.216608 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:28.216525 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 18 21:15:28.418097 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:28.418044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert\") pod \"authorino-6c44d9f6cf-6zs5s\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:28.420522 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:28.420503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert\") pod \"authorino-6c44d9f6cf-6zs5s\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:28.574715 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:28.574681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:15:28.856158 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:28.856077 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c44d9f6cf-6zs5s"] Sep 18 21:15:28.859133 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:15:28.859103 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd579e5cb_b61b_424e_9869_63d8f45ed068.slice/crio-23fdd785427c7475d89abf6e9ccfd7e6398a5a7de3db4d8c050476fd68108016 WatchSource:0}: Error finding container 23fdd785427c7475d89abf6e9ccfd7e6398a5a7de3db4d8c050476fd68108016: Status 404 returned error can't find the container with id 23fdd785427c7475d89abf6e9ccfd7e6398a5a7de3db4d8c050476fd68108016 Sep 18 21:15:29.084023 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:29.083989 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" event={"ID":"d579e5cb-b61b-424e-9869-63d8f45ed068","Type":"ContainerStarted","Data":"23fdd785427c7475d89abf6e9ccfd7e6398a5a7de3db4d8c050476fd68108016"} Sep 18 21:15:30.090730 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:30.090695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" event={"ID":"d579e5cb-b61b-424e-9869-63d8f45ed068","Type":"ContainerStarted","Data":"a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd"} Sep 18 21:15:30.110488 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:30.110431 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" podStartSLOduration=3.546085167 podStartE2EDuration="4.110414251s" totalImagesPullingTime="564.329084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:15:30.109427395 +0000 UTC m=+559.039173267" watchObservedRunningTime="2026-09-18 21:15:30.110414251 +0000 UTC m=+559.040160124" Sep 18 21:15:30.141058 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:30.141026 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2cxsk"] Sep 18 21:15:30.141232 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:30.141189 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2cxsk" podUID="0a455b9e-cc1d-4027-b6ef-e18052e575fb" containerName="authorino" containerID="cri-o://5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4" gracePeriod=30 Sep 18 21:15:30.399035 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:30.399012 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2cxsk" Sep 18 21:15:30.436220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:30.436189 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgtk4\" (UniqueName: \"kubernetes.io/projected/0a455b9e-cc1d-4027-b6ef-e18052e575fb-kube-api-access-lgtk4\") pod \"0a455b9e-cc1d-4027-b6ef-e18052e575fb\" (UID: \"0a455b9e-cc1d-4027-b6ef-e18052e575fb\") " Sep 18 21:15:30.438342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:30.438317 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a455b9e-cc1d-4027-b6ef-e18052e575fb-kube-api-access-lgtk4" (OuterVolumeSpecName: "kube-api-access-lgtk4") pod "0a455b9e-cc1d-4027-b6ef-e18052e575fb" (UID: "0a455b9e-cc1d-4027-b6ef-e18052e575fb"). InnerVolumeSpecName "kube-api-access-lgtk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:15:30.537127 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:30.537088 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgtk4\" (UniqueName: \"kubernetes.io/projected/0a455b9e-cc1d-4027-b6ef-e18052e575fb-kube-api-access-lgtk4\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:15:31.098548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.098512 2577 generic.go:356] "Generic (PLEG): container finished" podID="0a455b9e-cc1d-4027-b6ef-e18052e575fb" containerID="5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4" exitCode=0 Sep 18 21:15:31.098996 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.098552 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2cxsk" Sep 18 21:15:31.098996 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.098600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2cxsk" event={"ID":"0a455b9e-cc1d-4027-b6ef-e18052e575fb","Type":"ContainerDied","Data":"5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4"} Sep 18 21:15:31.098996 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.098640 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2cxsk" event={"ID":"0a455b9e-cc1d-4027-b6ef-e18052e575fb","Type":"ContainerDied","Data":"680778256d6a83e679a30a446e2483241dea6863bfd05828e7215c73d00b8cc1"} Sep 18 21:15:31.098996 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.098661 2577 scope.go:117] "RemoveContainer" containerID="5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4" Sep 18 21:15:31.110822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.110793 2577 scope.go:117] "RemoveContainer" containerID="5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4" Sep 18 21:15:31.111283 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:15:31.111256 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4\": container with ID starting with 5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4 not found: ID does not exist" containerID="5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4" Sep 18 21:15:31.111360 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.111296 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4"} err="failed to get container status \"5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4\": rpc error: code = NotFound desc = could not find container \"5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4\": container with ID starting with 5782389e5583c4aafa838b2fb3cd8c985f09cf9e8536175ab10ec1c8d68e74f4 not found: ID does not exist" Sep 18 21:15:31.133436 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.133399 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2cxsk"] Sep 18 21:15:31.140850 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.140806 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2cxsk"] Sep 18 21:15:31.359450 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:15:31.359353 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a455b9e-cc1d-4027-b6ef-e18052e575fb" path="/var/lib/kubelet/pods/0a455b9e-cc1d-4027-b6ef-e18052e575fb/volumes" Sep 18 21:16:13.105972 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.105926 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc"] Sep 18 21:16:13.106578 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.106560 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a455b9e-cc1d-4027-b6ef-e18052e575fb" containerName="authorino" Sep 18 21:16:13.106653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.106581 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a455b9e-cc1d-4027-b6ef-e18052e575fb" containerName="authorino" Sep 18 21:16:13.106704 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.106673 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a455b9e-cc1d-4027-b6ef-e18052e575fb" containerName="authorino" Sep 18 21:16:13.110329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.110306 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:13.112707 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.112683 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6kd5q\"" Sep 18 21:16:13.112828 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.112687 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 18 21:16:13.131185 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.131152 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc"] Sep 18 21:16:13.216797 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.216743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f9126a-2f1a-499c-9e9b-6c5e1854b31a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-nczzc\" (UID: \"b0f9126a-2f1a-499c-9e9b-6c5e1854b31a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:13.217004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.216920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ztn\" (UniqueName: \"kubernetes.io/projected/b0f9126a-2f1a-499c-9e9b-6c5e1854b31a-kube-api-access-g5ztn\") pod \"payload-pre-processing-59fbc47bf8-nczzc\" (UID: \"b0f9126a-2f1a-499c-9e9b-6c5e1854b31a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:13.306742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.306714 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:16:13.307204 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.307186 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:16:13.317667 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.317640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ztn\" (UniqueName: \"kubernetes.io/projected/b0f9126a-2f1a-499c-9e9b-6c5e1854b31a-kube-api-access-g5ztn\") pod \"payload-pre-processing-59fbc47bf8-nczzc\" (UID: \"b0f9126a-2f1a-499c-9e9b-6c5e1854b31a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:13.317780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.317760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f9126a-2f1a-499c-9e9b-6c5e1854b31a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-nczzc\" (UID: \"b0f9126a-2f1a-499c-9e9b-6c5e1854b31a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:13.321857 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.321830 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 18 21:16:13.328667 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.328645 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f9126a-2f1a-499c-9e9b-6c5e1854b31a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-nczzc\" (UID: \"b0f9126a-2f1a-499c-9e9b-6c5e1854b31a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:13.330138 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.330115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ztn\" (UniqueName: \"kubernetes.io/projected/b0f9126a-2f1a-499c-9e9b-6c5e1854b31a-kube-api-access-g5ztn\") pod \"payload-pre-processing-59fbc47bf8-nczzc\" (UID: \"b0f9126a-2f1a-499c-9e9b-6c5e1854b31a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:13.424730 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.424639 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6kd5q\"" Sep 18 21:16:13.432295 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.432267 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:13.734323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:13.734290 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc"] Sep 18 21:16:13.735013 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:16:13.734989 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f9126a_2f1a_499c_9e9b_6c5e1854b31a.slice/crio-5ec8077608809db150c4967d37ff020487bfe9f36fe02d21e5fb8daf7335c21f WatchSource:0}: Error finding container 5ec8077608809db150c4967d37ff020487bfe9f36fe02d21e5fb8daf7335c21f: Status 404 returned error can't find the container with id 5ec8077608809db150c4967d37ff020487bfe9f36fe02d21e5fb8daf7335c21f Sep 18 21:16:14.345783 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:14.345739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" event={"ID":"b0f9126a-2f1a-499c-9e9b-6c5e1854b31a","Type":"ContainerStarted","Data":"5ec8077608809db150c4967d37ff020487bfe9f36fe02d21e5fb8daf7335c21f"} Sep 18 21:16:16.365682 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:16.365640 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" event={"ID":"b0f9126a-2f1a-499c-9e9b-6c5e1854b31a","Type":"ContainerStarted","Data":"8c55e558f3d55d9a47b870c506e53eb7de34ffacac16ef3be5f48963ec5e1105"} Sep 18 21:16:16.368095 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:16.365700 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:16.392123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:16.392068 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" podStartSLOduration=1.748035401 podStartE2EDuration="3.392052211s" totalImagesPullingTime="1.64401681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:16:16.390037344 +0000 UTC m=+605.319783217" watchObservedRunningTime="2026-09-18 21:16:16.392052211 +0000 UTC m=+605.321798083" Sep 18 21:16:22.401751 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:22.401661 2577 generic.go:356] "Generic (PLEG): container finished" podID="0da2b319-54e0-4adc-928c-cf7faf5849be" containerID="aeb374f3175747b50d91d9dd2feb37f8771add6bde0afd0bd204b3a154d1ac87" exitCode=0 Sep 18 21:16:22.401751 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:22.401733 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" event={"ID":"0da2b319-54e0-4adc-928c-cf7faf5849be","Type":"ContainerDied","Data":"aeb374f3175747b50d91d9dd2feb37f8771add6bde0afd0bd204b3a154d1ac87"} Sep 18 21:16:22.402212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:22.402201 2577 scope.go:117] "RemoveContainer" containerID="aeb374f3175747b50d91d9dd2feb37f8771add6bde0afd0bd204b3a154d1ac87" Sep 18 21:16:22.403297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:22.403267 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:16:22.403405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:22.403340 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:16:22.403405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:22.403384 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:16:23.409262 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:23.409231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" event={"ID":"0da2b319-54e0-4adc-928c-cf7faf5849be","Type":"ContainerStarted","Data":"55520f98360861c752ca6c0d93cbcceeea57cff6937502fb4ee9832958e8550f"} Sep 18 21:16:24.218586 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:24.218552 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:16:24.218876 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:24.218849 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j9d9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 18 21:16:24.219032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:24.218900 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" podUID="0da2b319-54e0-4adc-928c-cf7faf5849be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 18 21:16:25.219257 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:25.219222 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j9d9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 18 21:16:25.219671 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:25.219279 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" podUID="0da2b319-54e0-4adc-928c-cf7faf5849be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 18 21:16:26.218902 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:26.218868 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j9d9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 18 21:16:26.219144 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:26.218946 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" podUID="0da2b319-54e0-4adc-928c-cf7faf5849be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 18 21:16:27.218836 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:27.218797 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j9d9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 18 21:16:27.219270 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:27.218856 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" podUID="0da2b319-54e0-4adc-928c-cf7faf5849be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 18 21:16:27.373136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:27.373104 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nczzc" Sep 18 21:16:28.218908 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:28.218867 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j9d9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 18 21:16:28.219348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:28.218934 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" podUID="0da2b319-54e0-4adc-928c-cf7faf5849be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 18 21:16:29.223326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:29.223294 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:16:29.223701 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:29.223570 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:16:29.224231 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:29.224215 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j9d9x" Sep 18 21:16:42.390603 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.390569 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf"] Sep 18 21:16:42.393430 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.393410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.396788 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.396764 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 18 21:16:42.396907 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.396785 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 18 21:16:42.396907 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.396889 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gv5wt\"" Sep 18 21:16:42.397031 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.396946 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 18 21:16:42.407351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.407332 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf"] Sep 18 21:16:42.413423 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.413401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.413546 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.413531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e887449e-0489-42c4-860a-216547d2a932-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.413620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.413592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.413706 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.413681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.413772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.413730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59zcn\" (UniqueName: \"kubernetes.io/projected/e887449e-0489-42c4-860a-216547d2a932-kube-api-access-59zcn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.413945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.413923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.516894 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.516858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.517107 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.516942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.517107 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.517097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e887449e-0489-42c4-860a-216547d2a932-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.517260 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.517127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.517315 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.517292 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.517372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.517347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.517427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.517402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zcn\" (UniqueName: \"kubernetes.io/projected/e887449e-0489-42c4-860a-216547d2a932-kube-api-access-59zcn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.517557 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.517537 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.517774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.517755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.519716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.519689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e887449e-0489-42c4-860a-216547d2a932-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.519853 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.519834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e887449e-0489-42c4-860a-216547d2a932-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.526840 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.526819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zcn\" (UniqueName: \"kubernetes.io/projected/e887449e-0489-42c4-860a-216547d2a932-kube-api-access-59zcn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-95vwf\" (UID: \"e887449e-0489-42c4-860a-216547d2a932\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.704223 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.704134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:42.976662 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.976633 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf"] Sep 18 21:16:42.978377 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:16:42.978342 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode887449e_0489_42c4_860a_216547d2a932.slice/crio-2b60046c2c5f9332e943b3b1e625f396aa3b9916002b0cbadd0f81c761814f0a WatchSource:0}: Error finding container 2b60046c2c5f9332e943b3b1e625f396aa3b9916002b0cbadd0f81c761814f0a: Status 404 returned error can't find the container with id 2b60046c2c5f9332e943b3b1e625f396aa3b9916002b0cbadd0f81c761814f0a Sep 18 21:16:42.980225 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:42.980206 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 21:16:43.544563 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:43.544522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" event={"ID":"e887449e-0489-42c4-860a-216547d2a932","Type":"ContainerStarted","Data":"2b60046c2c5f9332e943b3b1e625f396aa3b9916002b0cbadd0f81c761814f0a"} Sep 18 21:16:44.551644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:44.551605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" event={"ID":"e887449e-0489-42c4-860a-216547d2a932","Type":"ContainerStarted","Data":"444cceaa33fe36b5d4e4302f94e23a64795b3bcc128d58bbff236f031ebe152c"} Sep 18 21:16:44.570351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:44.570297 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" podStartSLOduration=1.5120179409999999 podStartE2EDuration="2.570282273s" totalImagesPullingTime="1.058264332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:16:44.56804828 +0000 UTC m=+633.497794157" watchObservedRunningTime="2026-09-18 21:16:44.570282273 +0000 UTC m=+633.500028145" Sep 18 21:16:52.705018 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:52.704976 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:52.705424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:52.705061 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:52.717559 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:52.717531 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:53.618033 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:53.617999 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-95vwf" Sep 18 21:16:58.888870 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.888837 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls"] Sep 18 21:16:58.892397 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.892377 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:58.898249 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.898222 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 18 21:16:58.917092 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.917059 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls"] Sep 18 21:16:58.966941 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.966902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d4cbc-63e5-4583-9bfb-1349a0448366-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:58.967155 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.966993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:58.967155 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.967022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:58.967155 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.967049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:58.967155 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.967126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzpv\" (UniqueName: \"kubernetes.io/projected/197d4cbc-63e5-4583-9bfb-1349a0448366-kube-api-access-qqzpv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:58.967292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:58.967169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068054 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d4cbc-63e5-4583-9bfb-1349a0448366-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzpv\" (UniqueName: \"kubernetes.io/projected/197d4cbc-63e5-4583-9bfb-1349a0448366-kube-api-access-qqzpv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068748 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.068937 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.068917 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.070649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.070627 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/197d4cbc-63e5-4583-9bfb-1349a0448366-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.071054 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.071031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/197d4cbc-63e5-4583-9bfb-1349a0448366-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.077727 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.077688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzpv\" (UniqueName: \"kubernetes.io/projected/197d4cbc-63e5-4583-9bfb-1349a0448366-kube-api-access-qqzpv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls\" (UID: \"197d4cbc-63e5-4583-9bfb-1349a0448366\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.203822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.203733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:16:59.481339 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.481308 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls"] Sep 18 21:16:59.481647 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:16:59.481625 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197d4cbc_63e5_4583_9bfb_1349a0448366.slice/crio-8eb3cdc909959736d10beee6b3a000870a9d23e7c2d4869df8d0dd96b41588f8 WatchSource:0}: Error finding container 8eb3cdc909959736d10beee6b3a000870a9d23e7c2d4869df8d0dd96b41588f8: Status 404 returned error can't find the container with id 8eb3cdc909959736d10beee6b3a000870a9d23e7c2d4869df8d0dd96b41588f8 Sep 18 21:16:59.641913 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:16:59.641872 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" event={"ID":"197d4cbc-63e5-4583-9bfb-1349a0448366","Type":"ContainerStarted","Data":"8eb3cdc909959736d10beee6b3a000870a9d23e7c2d4869df8d0dd96b41588f8"} Sep 18 21:17:00.651643 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:00.651595 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" event={"ID":"197d4cbc-63e5-4583-9bfb-1349a0448366","Type":"ContainerStarted","Data":"247374dab1b53c2ca6377f0883a8ccaa45a446658ff663e093dc53d12b9f37a2"} Sep 18 21:17:00.674430 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:00.674357 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" podStartSLOduration=2.495200384 podStartE2EDuration="2.674330405s" totalImagesPullingTime="179.130021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:17:00.670023682 +0000 UTC m=+649.599769555" watchObservedRunningTime="2026-09-18 21:17:00.674330405 +0000 UTC m=+649.604076279" Sep 18 21:17:09.204146 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:09.204107 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:17:09.204550 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:09.204175 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:17:09.216408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:09.216384 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:17:09.712880 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:09.712853 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls" Sep 18 21:17:10.702948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.702915 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj"] Sep 18 21:17:10.706676 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.706653 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.709440 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.709419 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 18 21:17:10.720497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.720472 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj"] Sep 18 21:17:10.785316 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.785287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.785451 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.785387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6wm\" (UniqueName: \"kubernetes.io/projected/60cb8262-6bab-4891-b050-05fc68eac5e7-kube-api-access-wc6wm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.785506 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.785457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.785506 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.785489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.785604 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.785573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb8262-6bab-4891-b050-05fc68eac5e7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.785668 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.785654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887204 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6wm\" (UniqueName: \"kubernetes.io/projected/60cb8262-6bab-4891-b050-05fc68eac5e7-kube-api-access-wc6wm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb8262-6bab-4891-b050-05fc68eac5e7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887699 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887765 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887743 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.887823 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.887767 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.889576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.889555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60cb8262-6bab-4891-b050-05fc68eac5e7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.889840 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.889823 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb8262-6bab-4891-b050-05fc68eac5e7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:10.895974 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:10.895935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6wm\" (UniqueName: \"kubernetes.io/projected/60cb8262-6bab-4891-b050-05fc68eac5e7-kube-api-access-wc6wm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2skvj\" (UID: \"60cb8262-6bab-4891-b050-05fc68eac5e7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:11.016918 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:11.016817 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:11.294862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:11.294831 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj"] Sep 18 21:17:11.296124 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:17:11.296096 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cb8262_6bab_4891_b050_05fc68eac5e7.slice/crio-b6d4d98fe1c78ccb5f1da11e187cc409957cc9a4444f93c4f5a2fbee44be29ab WatchSource:0}: Error finding container b6d4d98fe1c78ccb5f1da11e187cc409957cc9a4444f93c4f5a2fbee44be29ab: Status 404 returned error can't find the container with id b6d4d98fe1c78ccb5f1da11e187cc409957cc9a4444f93c4f5a2fbee44be29ab Sep 18 21:17:11.715292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:11.715248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" event={"ID":"60cb8262-6bab-4891-b050-05fc68eac5e7","Type":"ContainerStarted","Data":"2faee0923e64121262a49646aaf5a61d9e864bbb9575c79be9c66a361ee97ef5"} Sep 18 21:17:11.715661 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:11.715312 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" event={"ID":"60cb8262-6bab-4891-b050-05fc68eac5e7","Type":"ContainerStarted","Data":"b6d4d98fe1c78ccb5f1da11e187cc409957cc9a4444f93c4f5a2fbee44be29ab"} Sep 18 21:17:11.742383 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:11.742331 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" podStartSLOduration=1.5553163300000001 podStartE2EDuration="1.742315112s" totalImagesPullingTime="186.998782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:17:11.739837178 +0000 UTC m=+660.669583051" watchObservedRunningTime="2026-09-18 21:17:11.742315112 +0000 UTC m=+660.672061030" Sep 18 21:17:21.018004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:21.017937 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:21.018578 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:21.018037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:21.031030 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:21.030999 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:17:21.785542 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:17:21.785514 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2skvj" Sep 18 21:19:16.104126 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.104093 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-ffdcd95db-jqtqz"] Sep 18 21:19:16.107453 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.107436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ffdcd95db-jqtqz" Sep 18 21:19:16.127136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.127106 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ffdcd95db-jqtqz"] Sep 18 21:19:16.296300 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.296260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1def2918-6cf0-4249-9895-682e1e78ad78-tls-cert\") pod \"authorino-ffdcd95db-jqtqz\" (UID: \"1def2918-6cf0-4249-9895-682e1e78ad78\") " pod="kuadrant-system/authorino-ffdcd95db-jqtqz" Sep 18 21:19:16.296491 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.296357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5v6\" (UniqueName: \"kubernetes.io/projected/1def2918-6cf0-4249-9895-682e1e78ad78-kube-api-access-fg5v6\") pod \"authorino-ffdcd95db-jqtqz\" (UID: \"1def2918-6cf0-4249-9895-682e1e78ad78\") " pod="kuadrant-system/authorino-ffdcd95db-jqtqz" Sep 18 21:19:16.397199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.397098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1def2918-6cf0-4249-9895-682e1e78ad78-tls-cert\") pod \"authorino-ffdcd95db-jqtqz\" (UID: \"1def2918-6cf0-4249-9895-682e1e78ad78\") " pod="kuadrant-system/authorino-ffdcd95db-jqtqz" Sep 18 21:19:16.397199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.397186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5v6\" (UniqueName: \"kubernetes.io/projected/1def2918-6cf0-4249-9895-682e1e78ad78-kube-api-access-fg5v6\") pod \"authorino-ffdcd95db-jqtqz\" (UID: \"1def2918-6cf0-4249-9895-682e1e78ad78\") " pod="kuadrant-system/authorino-ffdcd95db-jqtqz" Sep 18 21:19:16.399622 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.399601 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1def2918-6cf0-4249-9895-682e1e78ad78-tls-cert\") pod \"authorino-ffdcd95db-jqtqz\" (UID: \"1def2918-6cf0-4249-9895-682e1e78ad78\") " pod="kuadrant-system/authorino-ffdcd95db-jqtqz" Sep 18 21:19:16.410536 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.410515 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5v6\" (UniqueName: \"kubernetes.io/projected/1def2918-6cf0-4249-9895-682e1e78ad78-kube-api-access-fg5v6\") pod \"authorino-ffdcd95db-jqtqz\" (UID: \"1def2918-6cf0-4249-9895-682e1e78ad78\") " pod="kuadrant-system/authorino-ffdcd95db-jqtqz" Sep 18 21:19:16.416315 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.416299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ffdcd95db-jqtqz" Sep 18 21:19:16.713986 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:16.713945 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ffdcd95db-jqtqz"] Sep 18 21:19:16.715624 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:19:16.715600 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1def2918_6cf0_4249_9895_682e1e78ad78.slice/crio-d5adffa2951c75c8f38b83a6c501718f29aaef2ce9cd16311b67a989e9cd1016 WatchSource:0}: Error finding container d5adffa2951c75c8f38b83a6c501718f29aaef2ce9cd16311b67a989e9cd1016: Status 404 returned error can't find the container with id d5adffa2951c75c8f38b83a6c501718f29aaef2ce9cd16311b67a989e9cd1016 Sep 18 21:19:17.457341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:17.457311 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ffdcd95db-jqtqz" event={"ID":"1def2918-6cf0-4249-9895-682e1e78ad78","Type":"ContainerStarted","Data":"d5adffa2951c75c8f38b83a6c501718f29aaef2ce9cd16311b67a989e9cd1016"} Sep 18 21:19:18.467777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.467742 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ffdcd95db-jqtqz" event={"ID":"1def2918-6cf0-4249-9895-682e1e78ad78","Type":"ContainerStarted","Data":"d4eae19b431938e4bde08937e4f592956380b5f7ec4747a8c3ee80f3bed7e710"} Sep 18 21:19:18.507737 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.507684 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-ffdcd95db-jqtqz" podStartSLOduration=1.8296952800000001 podStartE2EDuration="2.507666891s" totalImagesPullingTime="677.971611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:18.505005996 +0000 UTC m=+787.434751869" watchObservedRunningTime="2026-09-18 21:19:18.507666891 +0000 UTC m=+787.437412763" Sep 18 21:19:18.571039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.571008 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c44d9f6cf-6zs5s"] Sep 18 21:19:18.571227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.571188 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" podUID="d579e5cb-b61b-424e-9869-63d8f45ed068" containerName="authorino" containerID="cri-o://a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd" gracePeriod=30 Sep 18 21:19:18.827895 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.827852 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:19:18.917871 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.917838 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhpjg\" (UniqueName: \"kubernetes.io/projected/d579e5cb-b61b-424e-9869-63d8f45ed068-kube-api-access-qhpjg\") pod \"d579e5cb-b61b-424e-9869-63d8f45ed068\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " Sep 18 21:19:18.918071 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.917927 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert\") pod \"d579e5cb-b61b-424e-9869-63d8f45ed068\" (UID: \"d579e5cb-b61b-424e-9869-63d8f45ed068\") " Sep 18 21:19:18.920021 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.919995 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d579e5cb-b61b-424e-9869-63d8f45ed068-kube-api-access-qhpjg" (OuterVolumeSpecName: "kube-api-access-qhpjg") pod "d579e5cb-b61b-424e-9869-63d8f45ed068" (UID: "d579e5cb-b61b-424e-9869-63d8f45ed068"). InnerVolumeSpecName "kube-api-access-qhpjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:19:18.931282 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:18.931256 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d579e5cb-b61b-424e-9869-63d8f45ed068" (UID: "d579e5cb-b61b-424e-9869-63d8f45ed068"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:19:19.018796 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.018705 2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d579e5cb-b61b-424e-9869-63d8f45ed068-tls-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:19:19.018796 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.018748 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhpjg\" (UniqueName: \"kubernetes.io/projected/d579e5cb-b61b-424e-9869-63d8f45ed068-kube-api-access-qhpjg\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:19:19.475212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.475171 2577 generic.go:356] "Generic (PLEG): container finished" podID="d579e5cb-b61b-424e-9869-63d8f45ed068" containerID="a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd" exitCode=0 Sep 18 21:19:19.475212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.475216 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" Sep 18 21:19:19.475751 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.475257 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" event={"ID":"d579e5cb-b61b-424e-9869-63d8f45ed068","Type":"ContainerDied","Data":"a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd"} Sep 18 21:19:19.475751 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.475314 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c44d9f6cf-6zs5s" event={"ID":"d579e5cb-b61b-424e-9869-63d8f45ed068","Type":"ContainerDied","Data":"23fdd785427c7475d89abf6e9ccfd7e6398a5a7de3db4d8c050476fd68108016"} Sep 18 21:19:19.475751 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.475337 2577 scope.go:117] "RemoveContainer" containerID="a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd" Sep 18 21:19:19.488044 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.488020 2577 scope.go:117] "RemoveContainer" containerID="a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd" Sep 18 21:19:19.488394 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:19:19.488364 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd\": container with ID starting with a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd not found: ID does not exist" containerID="a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd" Sep 18 21:19:19.488470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.488407 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd"} err="failed to get container status \"a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd\": rpc error: code = NotFound desc = could not find container \"a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd\": container with ID starting with a9bac1b2e4ceb3f2aa57b02e7bc623e93060fab542a9d100971c6619e1e931cd not found: ID does not exist" Sep 18 21:19:19.687805 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.687714 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c44d9f6cf-6zs5s"] Sep 18 21:19:19.696920 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:19.696892 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c44d9f6cf-6zs5s"] Sep 18 21:19:21.359724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:21.359686 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d579e5cb-b61b-424e-9869-63d8f45ed068" path="/var/lib/kubelet/pods/d579e5cb-b61b-424e-9869-63d8f45ed068/volumes" Sep 18 21:19:37.387990 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.387892 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55"] Sep 18 21:19:37.388352 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.388317 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d579e5cb-b61b-424e-9869-63d8f45ed068" containerName="authorino" Sep 18 21:19:37.388352 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.388329 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d579e5cb-b61b-424e-9869-63d8f45ed068" containerName="authorino" Sep 18 21:19:37.388425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.388397 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d579e5cb-b61b-424e-9869-63d8f45ed068" containerName="authorino" Sep 18 21:19:37.391392 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.391367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.400285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.400267 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-116f55-openshift-default-dockercfg-74968\"" Sep 18 21:19:37.435669 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.435631 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55"] Sep 18 21:19:37.457196 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-envoy\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.457391 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istiod-ca-cert\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.457391 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-podinfo\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.457391 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-data\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.457391 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-credential-socket\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.457555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-socket\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.457555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-certs\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.457555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-token\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.457678 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.457579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp85w\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-kube-api-access-lp85w\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.552604 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.552570 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2"] Sep 18 21:19:37.556159 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.556141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.558728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.558696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-credential-socket\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.558858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.558743 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6ad49b-openshift-default-dockercfg-fvc8d\"" Sep 18 21:19:37.558858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.558781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-socket\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.558858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.558823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-certs\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.558870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-token\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.558924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp85w\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-kube-api-access-lp85w\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.558989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-envoy\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.559041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istiod-ca-cert\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.559100 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-credential-socket\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.559156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-podinfo\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.559168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-socket\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.559188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-data\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559452 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.559203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-certs\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559510 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.559463 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-data\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.559714 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.559693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istiod-ca-cert\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.561404 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.561370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-envoy\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.561577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.561558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-podinfo\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.582527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.582493 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp85w\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-kube-api-access-lp85w\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.582756 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.582736 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2"] Sep 18 21:19:37.584050 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.584023 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-token\") pod \"e2e-inf-a-116f55-openshift-default-668f5789c8-frx55\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.660329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-envoy\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.660329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-certs\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.660329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-data\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.660625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-podinfo\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.660625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca8d284-0164-444b-bb30-01745ccdf5df-istiod-ca-cert\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.660625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-socket\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.660625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2qs\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-kube-api-access-hm2qs\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.660625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-credential-socket\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.660906 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.660657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-token\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.702177 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.702143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.761630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2qs\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-kube-api-access-hm2qs\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.761683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-credential-socket\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.761748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-token\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.761776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-envoy\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.761803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-certs\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.761824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-data\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.761876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-podinfo\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.761945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca8d284-0164-444b-bb30-01745ccdf5df-istiod-ca-cert\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.762251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.762060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-socket\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.764136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.763231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-credential-socket\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.764136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.763598 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-certs\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.764136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.763862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca8d284-0164-444b-bb30-01745ccdf5df-istiod-ca-cert\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.764499 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.764281 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-data\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.764591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.764565 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-socket\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.766559 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.766532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-podinfo\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.767557 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.767529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-envoy\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.773855 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.773804 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-token\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.774028 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.773889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2qs\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-kube-api-access-hm2qs\") pod \"e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.878053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.878023 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:37.996905 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.996876 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55"] Sep 18 21:19:37.997056 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:19:37.996869 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d92aaa_6964_43b3_b0e2_0d52c1e9c97b.slice/crio-d1ff4c5a03bb316858bf35e1f078f502736204137849b3334e7a97b7293b8950 WatchSource:0}: Error finding container d1ff4c5a03bb316858bf35e1f078f502736204137849b3334e7a97b7293b8950: Status 404 returned error can't find the container with id d1ff4c5a03bb316858bf35e1f078f502736204137849b3334e7a97b7293b8950 Sep 18 21:19:37.999468 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.999434 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:37.999587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.999517 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:37.999587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:37.999559 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:38.176980 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.176921 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2"] Sep 18 21:19:38.177251 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:19:38.177226 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca8d284_0164_444b_bb30_01745ccdf5df.slice/crio-6b44ec2c1485b442af8df8d5162365bfdad97e1e2b634c44d6688a7131a21d59 WatchSource:0}: Error finding container 6b44ec2c1485b442af8df8d5162365bfdad97e1e2b634c44d6688a7131a21d59: Status 404 returned error can't find the container with id 6b44ec2c1485b442af8df8d5162365bfdad97e1e2b634c44d6688a7131a21d59 Sep 18 21:19:38.180057 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.180023 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:38.180167 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.180103 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:38.180167 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.180133 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:38.586359 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.586321 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" event={"ID":"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b","Type":"ContainerStarted","Data":"e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39"} Sep 18 21:19:38.586791 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.586388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" event={"ID":"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b","Type":"ContainerStarted","Data":"d1ff4c5a03bb316858bf35e1f078f502736204137849b3334e7a97b7293b8950"} Sep 18 21:19:38.588046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.588021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" event={"ID":"1ca8d284-0164-444b-bb30-01745ccdf5df","Type":"ContainerStarted","Data":"bc0db5dbdd5ff073062ac6efd145a9dae30f61bfcaecfbb97037203ae178162c"} Sep 18 21:19:38.588160 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.588076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" event={"ID":"1ca8d284-0164-444b-bb30-01745ccdf5df","Type":"ContainerStarted","Data":"6b44ec2c1485b442af8df8d5162365bfdad97e1e2b634c44d6688a7131a21d59"} Sep 18 21:19:38.623005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.622930 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" podStartSLOduration=1.62291465 podStartE2EDuration="1.62291465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:38.620897371 +0000 UTC m=+807.550643258" watchObservedRunningTime="2026-09-18 21:19:38.62291465 +0000 UTC m=+807.552660589" Sep 18 21:19:38.648125 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.648062 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" podStartSLOduration=1.648045357 podStartE2EDuration="1.648045357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:38.646543761 +0000 UTC m=+807.576289637" watchObservedRunningTime="2026-09-18 21:19:38.648045357 +0000 UTC m=+807.577791328" Sep 18 21:19:38.702571 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.702533 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:38.878889 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:38.878798 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:39.707805 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:39.707768 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:39.883980 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:39.883922 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:40.600789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:40.600758 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:40.601032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:40.600820 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:40.601797 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:40.601765 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:19:40.601907 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:40.601850 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:19:45.279742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.279710 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb"] Sep 18 21:19:45.284491 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.284473 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.290636 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.290615 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-21df59-openshift-default-dockercfg-f2mng\"" Sep 18 21:19:45.342337 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-data\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.342499 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-credential-socket\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.342499 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-certs\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.342499 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnx48\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-kube-api-access-bnx48\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.342612 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-socket\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.342612 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-envoy\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.342612 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cd998d-fbdc-4217-a92b-0abeb6151e24-istiod-ca-cert\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.342709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-token\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.342709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.342668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-podinfo\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-socket\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-envoy\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cd998d-fbdc-4217-a92b-0abeb6151e24-istiod-ca-cert\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-token\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-podinfo\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-data\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-credential-socket\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-certs\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-credential-socket\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnx48\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-kube-api-access-bnx48\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.444790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.444771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-data\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.445111 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.445057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-certs\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.445278 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.445226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-socket\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.445278 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.445240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cd998d-fbdc-4217-a92b-0abeb6151e24-istiod-ca-cert\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.447184 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.447157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-envoy\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.447443 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.447425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-podinfo\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.466474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.466437 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb"] Sep 18 21:19:45.480457 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.480424 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnx48\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-kube-api-access-bnx48\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.493291 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.493260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-token\") pod \"e2e-autores-21df59-openshift-default-7775879dcb-8q4vb\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.595190 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.595157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:45.981148 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.981123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb"] Sep 18 21:19:45.982522 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:19:45.982496 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cd998d_fbdc_4217_a92b_0abeb6151e24.slice/crio-03b00115f906eb0c5c7e6e03c96af9ca14f8933d811c88e9851e80c27346b350 WatchSource:0}: Error finding container 03b00115f906eb0c5c7e6e03c96af9ca14f8933d811c88e9851e80c27346b350: Status 404 returned error can't find the container with id 03b00115f906eb0c5c7e6e03c96af9ca14f8933d811c88e9851e80c27346b350 Sep 18 21:19:45.984735 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.984704 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:45.984819 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.984769 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:45.984819 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:45.984797 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:46.644549 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:46.644508 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" event={"ID":"18cd998d-fbdc-4217-a92b-0abeb6151e24","Type":"ContainerStarted","Data":"365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7"} Sep 18 21:19:46.645110 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:46.644790 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" event={"ID":"18cd998d-fbdc-4217-a92b-0abeb6151e24","Type":"ContainerStarted","Data":"03b00115f906eb0c5c7e6e03c96af9ca14f8933d811c88e9851e80c27346b350"} Sep 18 21:19:46.768004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:46.767920 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podStartSLOduration=1.7679021179999999 podStartE2EDuration="1.767902118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:46.767495711 +0000 UTC m=+815.697241584" watchObservedRunningTime="2026-09-18 21:19:46.767902118 +0000 UTC m=+815.697647991" Sep 18 21:19:47.595802 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:47.595765 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:47.606221 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:47.606175 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:47.651359 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:47.651330 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:47.652427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:47.652399 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:19:53.875133 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:53.875081 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5"] Sep 18 21:19:53.878946 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:53.878913 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:53.883050 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:53.883025 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ef01a1fd-openshift-default-dockercfg-ftxc7\"" Sep 18 21:19:53.966097 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:53.966061 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5"] Sep 18 21:19:54.039617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.039583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-socket\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.039617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.039627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-token\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.039890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.039655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-data\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.039890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.039732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825b7242-dd7e-420a-b67b-3a7975331cb4-istiod-ca-cert\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.039890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.039819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-credential-socket\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.039890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.039875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-envoy\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.040075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.039932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-podinfo\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.040075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.040061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-certs\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.040152 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.040088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994kx\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-kube-api-access-994kx\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141517 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-podinfo\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-certs\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994kx\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-kube-api-access-994kx\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-socket\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-token\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-data\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141987 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825b7242-dd7e-420a-b67b-3a7975331cb4-istiod-ca-cert\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141987 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-credential-socket\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.141987 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.141867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-envoy\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.144800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.142702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-socket\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.144800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.143351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-certs\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.144800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.143587 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-data\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.144800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.144790 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825b7242-dd7e-420a-b67b-3a7975331cb4-istiod-ca-cert\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.157975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.145127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-credential-socket\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.157975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.145398 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-envoy\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.157975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.145940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-podinfo\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.167003 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.166970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-994kx\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-kube-api-access-994kx\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.178240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.178209 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-token\") pod \"e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.191307 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.191280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:54.558494 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.558459 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5"] Sep 18 21:19:54.559625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.559580 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:54.559720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.559680 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:54.559783 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.559729 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:54.696526 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.696489 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" event={"ID":"825b7242-dd7e-420a-b67b-3a7975331cb4","Type":"ContainerStarted","Data":"f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb"} Sep 18 21:19:54.696741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:54.696545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" event={"ID":"825b7242-dd7e-420a-b67b-3a7975331cb4","Type":"ContainerStarted","Data":"a60a4afa16cee5307b01776444edd14152b4e53340f9bdac7ab62b4ea264fdf2"} Sep 18 21:19:55.192580 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:55.192544 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:55.194382 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:55.194345 2577 patch_prober.go:28] interesting pod/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 18 21:19:55.194531 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:55.194414 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" podUID="825b7242-dd7e-420a-b67b-3a7975331cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 18 21:19:56.198567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.198014 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:56.198567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.198324 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" podStartSLOduration=3.198303317 podStartE2EDuration="3.198303317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:54.766790441 +0000 UTC m=+823.696536316" watchObservedRunningTime="2026-09-18 21:19:56.198303317 +0000 UTC m=+825.128049191" Sep 18 21:19:56.201183 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.201156 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4"] Sep 18 21:19:56.205319 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.205296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.209067 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.209029 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1f86a8-openshift-default-dockercfg-kp57z\"" Sep 18 21:19:56.235139 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.235104 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4"] Sep 18 21:19:56.389999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.389739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-envoy\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.389999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.389793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istiod-ca-cert\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.389999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.389824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-credential-socket\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.389999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.389875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-token\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.389999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.389923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-socket\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.390410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.390059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-podinfo\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.390410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.390099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-certs\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.390410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.390127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6zg\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-kube-api-access-2q6zg\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.390410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.390170 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-data\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491327 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-envoy\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491327 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istiod-ca-cert\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491327 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-credential-socket\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491556 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-token\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491556 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-socket\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491556 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-podinfo\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491658 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-certs\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491658 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6zg\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-kube-api-access-2q6zg\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-data\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.491758 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.491735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-credential-socket\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.492091 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.492072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-socket\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.492140 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.492076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-data\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.492220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.492199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istiod-ca-cert\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.492402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.492365 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-certs\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.494197 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.494163 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-podinfo\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.494354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.494334 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-envoy\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.542981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.542743 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-token\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.553981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.551299 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6zg\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-kube-api-access-2q6zg\") pod \"e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:56.708301 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.708273 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:56.709216 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.709201 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:19:56.817305 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:56.817270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:57.187362 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:57.186949 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4"] Sep 18 21:19:57.196794 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:57.196397 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:57.196794 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:57.196516 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:57.196794 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:57.196576 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:19:57.719367 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:57.719329 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" event={"ID":"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336","Type":"ContainerStarted","Data":"5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783"} Sep 18 21:19:57.719860 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:57.719478 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" event={"ID":"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336","Type":"ContainerStarted","Data":"78b4f610c9490651c820170f63dd0e9799ea0a5949ab6f3ab28d86d21800c37b"} Sep 18 21:19:57.800975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:57.790541 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" podStartSLOduration=1.7905227369999999 podStartE2EDuration="1.790522737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:57.785902387 +0000 UTC m=+826.715648260" watchObservedRunningTime="2026-09-18 21:19:57.790522737 +0000 UTC m=+826.720268610" Sep 18 21:19:57.817678 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:57.817645 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:58.823913 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:58.823881 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:59.737453 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:59.737424 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:19:59.743752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:19:59.743702 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:20:03.775760 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.775727 2577 generic.go:356] "Generic (PLEG): container finished" podID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerID="e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39" exitCode=0 Sep 18 21:20:03.776216 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.775794 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" event={"ID":"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b","Type":"ContainerDied","Data":"e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39"} Sep 18 21:20:03.776385 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.776352 2577 scope.go:117] "RemoveContainer" containerID="e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39" Sep 18 21:20:03.777875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.777851 2577 generic.go:356] "Generic (PLEG): container finished" podID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerID="bc0db5dbdd5ff073062ac6efd145a9dae30f61bfcaecfbb97037203ae178162c" exitCode=0 Sep 18 21:20:03.777998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.777876 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" event={"ID":"1ca8d284-0164-444b-bb30-01745ccdf5df","Type":"ContainerDied","Data":"bc0db5dbdd5ff073062ac6efd145a9dae30f61bfcaecfbb97037203ae178162c"} Sep 18 21:20:03.777998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.777890 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:03.777998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.777969 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:03.778122 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.778008 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:03.778238 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.778224 2577 scope.go:117] "RemoveContainer" containerID="bc0db5dbdd5ff073062ac6efd145a9dae30f61bfcaecfbb97037203ae178162c" Sep 18 21:20:03.779686 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.779644 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:03.779779 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.779706 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:03.779779 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:03.779738 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:04.788653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:04.788619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" event={"ID":"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b","Type":"ContainerStarted","Data":"75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28"} Sep 18 21:20:04.790709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:04.790681 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" event={"ID":"1ca8d284-0164-444b-bb30-01745ccdf5df","Type":"ContainerStarted","Data":"2fc3a6969a7d6730e6b9a1874812320397255ef90bf9d5f75c088da5216720f2"} Sep 18 21:20:04.878716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:04.878674 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:20:04.879000 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:04.878973 2577 patch_prober.go:28] interesting pod/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 21:20:04.879111 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:04.879032 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 21:20:05.702695 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.702656 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:20:05.702888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.702762 2577 patch_prober.go:28] interesting pod/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 18 21:20:05.702888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.702796 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 18 21:20:05.877980 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.877908 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv"] Sep 18 21:20:05.881989 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.878836 2577 patch_prober.go:28] interesting pod/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 21:20:05.881989 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.878901 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 21:20:05.889993 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.889894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:05.892818 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.892795 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ef01a1fd-dockercfg-9wmzs\"" Sep 18 21:20:05.893127 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.893112 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ef01a1fd\"" Sep 18 21:20:05.907814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:05.907787 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv"] Sep 18 21:20:06.012307 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.012203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv\" (UID: \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\") " pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:06.012307 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.012275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpn9\" (UniqueName: \"kubernetes.io/projected/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-kube-api-access-8vpn9\") pod \"payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv\" (UID: \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\") " pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:06.113904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.113864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv\" (UID: \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\") " pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:06.114157 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.113927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpn9\" (UniqueName: \"kubernetes.io/projected/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-kube-api-access-8vpn9\") pod \"payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv\" (UID: \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\") " pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:06.114638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.114614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv\" (UID: \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\") " pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:06.125153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.125070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpn9\" (UniqueName: \"kubernetes.io/projected/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-kube-api-access-8vpn9\") pod \"payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv\" (UID: \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\") " pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:06.203877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.203837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:06.569056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.569012 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv"] Sep 18 21:20:06.580516 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:20:06.580467 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cf2fe0_d6df_4cf3_a357_5a4446fc89dd.slice/crio-3bbf9e2576be8ff28cfd1810f2b27efbe3c6404521d3446030d00ad99d4bd774 WatchSource:0}: Error finding container 3bbf9e2576be8ff28cfd1810f2b27efbe3c6404521d3446030d00ad99d4bd774: Status 404 returned error can't find the container with id 3bbf9e2576be8ff28cfd1810f2b27efbe3c6404521d3446030d00ad99d4bd774 Sep 18 21:20:06.702684 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.702654 2577 patch_prober.go:28] interesting pod/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 18 21:20:06.702811 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.702705 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 18 21:20:06.812469 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.812422 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" event={"ID":"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd","Type":"ContainerStarted","Data":"2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441"} Sep 18 21:20:06.812701 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.812502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" event={"ID":"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd","Type":"ContainerStarted","Data":"3bbf9e2576be8ff28cfd1810f2b27efbe3c6404521d3446030d00ad99d4bd774"} Sep 18 21:20:06.812886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.812861 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:06.864882 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.864747 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" podStartSLOduration=1.864726213 podStartE2EDuration="1.864726213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:20:06.861204145 +0000 UTC m=+835.790950021" watchObservedRunningTime="2026-09-18 21:20:06.864726213 +0000 UTC m=+835.794472086" Sep 18 21:20:06.879492 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.879447 2577 patch_prober.go:28] interesting pod/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 21:20:06.880022 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:06.879516 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 21:20:07.703157 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:07.703123 2577 patch_prober.go:28] interesting pod/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 18 21:20:07.703408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:07.703386 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 18 21:20:07.878703 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:07.878664 2577 patch_prober.go:28] interesting pod/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 21:20:07.878998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:07.878947 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 21:20:08.704099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:08.704057 2577 patch_prober.go:28] interesting pod/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 18 21:20:08.704617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:08.704133 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 18 21:20:08.878605 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:08.878569 2577 patch_prober.go:28] interesting pod/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 21:20:08.878905 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:08.878876 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 21:20:09.703620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:09.703575 2577 patch_prober.go:28] interesting pod/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 18 21:20:09.703842 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:09.703655 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 18 21:20:09.893574 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:09.893529 2577 patch_prober.go:28] interesting pod/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 21:20:09.894760 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:09.894725 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 21:20:10.601306 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.601257 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:20:10.601489 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.601341 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:20:10.706482 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.706446 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:20:10.759078 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.759037 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv"] Sep 18 21:20:10.762069 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.762041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:10.772705 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.772685 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1f86a8-dockercfg-x2wdh\"" Sep 18 21:20:10.775107 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.775089 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1f86a8\"" Sep 18 21:20:10.798027 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.797994 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv"] Sep 18 21:20:10.858612 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.858516 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:20:10.883754 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.883721 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:20:10.885046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.885024 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:20:10.917766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.917719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmrv\" (UniqueName: \"kubernetes.io/projected/9ae2d026-ca92-4faa-a789-e3930ac681e8-kube-api-access-dwmrv\") pod \"payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv\" (UID: \"9ae2d026-ca92-4faa-a789-e3930ac681e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:10.918239 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:10.917844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2d026-ca92-4faa-a789-e3930ac681e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv\" (UID: \"9ae2d026-ca92-4faa-a789-e3930ac681e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:11.018909 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.018860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmrv\" (UniqueName: \"kubernetes.io/projected/9ae2d026-ca92-4faa-a789-e3930ac681e8-kube-api-access-dwmrv\") pod \"payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv\" (UID: \"9ae2d026-ca92-4faa-a789-e3930ac681e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:11.019131 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.019030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2d026-ca92-4faa-a789-e3930ac681e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv\" (UID: \"9ae2d026-ca92-4faa-a789-e3930ac681e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:11.019772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.019749 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2d026-ca92-4faa-a789-e3930ac681e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv\" (UID: \"9ae2d026-ca92-4faa-a789-e3930ac681e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:11.067343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.067305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmrv\" (UniqueName: \"kubernetes.io/projected/9ae2d026-ca92-4faa-a789-e3930ac681e8-kube-api-access-dwmrv\") pod \"payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv\" (UID: \"9ae2d026-ca92-4faa-a789-e3930ac681e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:11.072143 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.072110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:11.404153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.404128 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv"] Sep 18 21:20:11.405745 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:20:11.405719 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae2d026_ca92_4faa_a789_e3930ac681e8.slice/crio-649b65000b78a58da4f041a50dfe2d25e31f77656947dc2587eba256f38318ee WatchSource:0}: Error finding container 649b65000b78a58da4f041a50dfe2d25e31f77656947dc2587eba256f38318ee: Status 404 returned error can't find the container with id 649b65000b78a58da4f041a50dfe2d25e31f77656947dc2587eba256f38318ee Sep 18 21:20:11.865229 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.865189 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" event={"ID":"9ae2d026-ca92-4faa-a789-e3930ac681e8","Type":"ContainerStarted","Data":"0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214"} Sep 18 21:20:11.865449 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.865257 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:11.865449 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.865280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" event={"ID":"9ae2d026-ca92-4faa-a789-e3930ac681e8","Type":"ContainerStarted","Data":"649b65000b78a58da4f041a50dfe2d25e31f77656947dc2587eba256f38318ee"} Sep 18 21:20:11.886618 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:11.886557 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" podStartSLOduration=1.886539209 podStartE2EDuration="1.886539209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:20:11.883636517 +0000 UTC m=+840.813382391" watchObservedRunningTime="2026-09-18 21:20:11.886539209 +0000 UTC m=+840.816285082" Sep 18 21:20:17.653026 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.652916 2577 patch_prober.go:28] interesting pod/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 18 21:20:17.653433 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.653002 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 18 21:20:17.824132 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.824104 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:17.906077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.905968 2577 generic.go:356] "Generic (PLEG): container finished" podID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerID="365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7" exitCode=0 Sep 18 21:20:17.906077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.905996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" event={"ID":"18cd998d-fbdc-4217-a92b-0abeb6151e24","Type":"ContainerDied","Data":"365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7"} Sep 18 21:20:17.906527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.906510 2577 scope.go:117] "RemoveContainer" containerID="365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7" Sep 18 21:20:17.907889 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.907857 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:17.908011 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.907941 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:17.908089 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:17.908014 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:18.914410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:18.914376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" event={"ID":"18cd998d-fbdc-4217-a92b-0abeb6151e24","Type":"ContainerStarted","Data":"42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f"} Sep 18 21:20:19.596273 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:19.596235 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:20:19.596527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:19.596495 2577 patch_prober.go:28] interesting pod/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 18 21:20:19.596663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:19.596554 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 18 21:20:20.596366 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:20.596329 2577 patch_prober.go:28] interesting pod/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 18 21:20:20.596763 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:20.596389 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 18 21:20:21.595717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:21.595678 2577 patch_prober.go:28] interesting pod/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 18 21:20:21.595912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:21.595733 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 18 21:20:22.595858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:22.595821 2577 patch_prober.go:28] interesting pod/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 18 21:20:22.596504 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:22.596453 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 18 21:20:22.874912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:22.874806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:20:23.596082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:23.596026 2577 patch_prober.go:28] interesting pod/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 18 21:20:23.597057 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:23.596719 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 18 21:20:24.596405 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:24.596356 2577 patch_prober.go:28] interesting pod/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 18 21:20:24.596978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:24.596431 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 18 21:20:25.599843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:25.599812 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:20:25.600529 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:25.600510 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:20:25.601100 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:25.601083 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:20:27.474899 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.474814 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv"] Sep 18 21:20:27.475924 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.475613 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" podUID="e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd" containerName="payload-processing" containerID="cri-o://2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441" gracePeriod=30 Sep 18 21:20:27.790083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.790058 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:27.879842 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.879809 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-plugins-config-volume\") pod \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\" (UID: \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\") " Sep 18 21:20:27.880038 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.879916 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vpn9\" (UniqueName: \"kubernetes.io/projected/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-kube-api-access-8vpn9\") pod \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\" (UID: \"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd\") " Sep 18 21:20:27.880178 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.880152 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd" (UID: "e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:20:27.880358 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.880338 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-plugins-config-volume\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:27.882118 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.882086 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-kube-api-access-8vpn9" (OuterVolumeSpecName: "kube-api-access-8vpn9") pod "e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd" (UID: "e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd"). InnerVolumeSpecName "kube-api-access-8vpn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:20:27.982979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:27.982689 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpn9\" (UniqueName: \"kubernetes.io/projected/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd-kube-api-access-8vpn9\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:28.023425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.023336 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5"] Sep 18 21:20:28.023604 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.023579 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" podUID="825b7242-dd7e-420a-b67b-3a7975331cb4" containerName="istio-proxy" containerID="cri-o://f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb" gracePeriod=30 Sep 18 21:20:28.047779 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.047746 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd" containerID="2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441" exitCode=0 Sep 18 21:20:28.047948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.047826 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" Sep 18 21:20:28.047948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.047857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" event={"ID":"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd","Type":"ContainerDied","Data":"2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441"} Sep 18 21:20:28.047948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.047929 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv" event={"ID":"e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd","Type":"ContainerDied","Data":"3bbf9e2576be8ff28cfd1810f2b27efbe3c6404521d3446030d00ad99d4bd774"} Sep 18 21:20:28.048102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.047983 2577 scope.go:117] "RemoveContainer" containerID="2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441" Sep 18 21:20:28.058639 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.058520 2577 scope.go:117] "RemoveContainer" containerID="2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441" Sep 18 21:20:28.058899 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:20:28.058877 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441\": container with ID starting with 2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441 not found: ID does not exist" containerID="2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441" Sep 18 21:20:28.058997 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.058912 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441"} err="failed to get container status \"2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441\": rpc error: code = NotFound desc = could not find container \"2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441\": container with ID starting with 2077ec47cf5605679370565321239811c08afd9e8680e1b2d7ba3a21d2b24441 not found: ID does not exist" Sep 18 21:20:28.134225 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.134177 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv"] Sep 18 21:20:28.146347 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:28.146311 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ef01a1fd-6d7bd565c5-tgzhv"] Sep 18 21:20:29.362497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.362457 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd" path="/var/lib/kubelet/pods/e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd/volumes" Sep 18 21:20:29.476767 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.476732 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp"] Sep 18 21:20:29.477236 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.477222 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd" containerName="payload-processing" Sep 18 21:20:29.477236 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.477238 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd" containerName="payload-processing" Sep 18 21:20:29.477341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.477302 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cf2fe0-d6df-4cf3-a357-5a4446fc89dd" containerName="payload-processing" Sep 18 21:20:29.479787 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.479771 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.486107 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.486081 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7e01a72f-gw-openshift-default-dockercfg-vm7qd\"" Sep 18 21:20:29.547113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.547072 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp"] Sep 18 21:20:29.601916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.601879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-data\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.602123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.601948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-socket\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.602123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.602026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-certs\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.602123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.602049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istiod-ca-cert\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.602251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.602169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnsq\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-kube-api-access-gsnsq\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.602316 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.602298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-credential-socket\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.602354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.602334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-envoy\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.602389 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.602368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-podinfo\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.602426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.602411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-token\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-token\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-data\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-socket\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-certs\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istiod-ca-cert\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnsq\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-kube-api-access-gsnsq\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703694 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-data\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703753 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-credential-socket\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703813 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-envoy\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.703813 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.703781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-podinfo\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.704174 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.704120 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-credential-socket\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.704301 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.704268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istiod-ca-cert\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.704426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.704406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-certs\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.704513 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.704452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-socket\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.706299 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.706279 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-envoy\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.706886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.706853 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-podinfo\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.716029 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.716001 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnsq\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-kube-api-access-gsnsq\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.716969 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.716934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-token\") pod \"e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:29.790645 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:29.790597 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:30.114548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:30.114512 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp"] Sep 18 21:20:30.116205 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:20:30.116177 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0ecd32_194b_45b8_bc8d_6a69ed55ef98.slice/crio-c8c9716ba7f679871069b6b2c4eb18c01a3c66324c01b9a4dd5825a6f97b9680 WatchSource:0}: Error finding container c8c9716ba7f679871069b6b2c4eb18c01a3c66324c01b9a4dd5825a6f97b9680: Status 404 returned error can't find the container with id c8c9716ba7f679871069b6b2c4eb18c01a3c66324c01b9a4dd5825a6f97b9680 Sep 18 21:20:30.119637 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:30.119593 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:30.119725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:30.119689 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:30.119774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:30.119735 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:30.739144 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:30.739004 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 18 21:20:30.739144 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:30.739068 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 21:20:31.072516 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.072473 2577 generic.go:356] "Generic (PLEG): container finished" podID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerID="5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783" exitCode=0 Sep 18 21:20:31.072719 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.072538 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" event={"ID":"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336","Type":"ContainerDied","Data":"5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783"} Sep 18 21:20:31.073109 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.073092 2577 scope.go:117] "RemoveContainer" containerID="5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783" Sep 18 21:20:31.074412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.074384 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:31.074497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.074455 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:31.074497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.074492 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:31.074741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.074710 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" event={"ID":"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98","Type":"ContainerStarted","Data":"d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3"} Sep 18 21:20:31.074866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.074767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" event={"ID":"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98","Type":"ContainerStarted","Data":"c8c9716ba7f679871069b6b2c4eb18c01a3c66324c01b9a4dd5825a6f97b9680"} Sep 18 21:20:31.259776 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.259716 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" podStartSLOduration=2.259695881 podStartE2EDuration="2.259695881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:20:31.250657633 +0000 UTC m=+860.180403509" watchObservedRunningTime="2026-09-18 21:20:31.259695881 +0000 UTC m=+860.189441821" Sep 18 21:20:31.790924 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.790827 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:31.796146 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.796112 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:31.854459 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.854304 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945331 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-994kx\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-kube-api-access-994kx\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945461 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-token\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945548 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-certs\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945585 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-socket\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945613 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825b7242-dd7e-420a-b67b-3a7975331cb4-istiod-ca-cert\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945640 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-credential-socket\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945666 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-podinfo\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945704 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-envoy\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.945979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.945767 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-data\") pod \"825b7242-dd7e-420a-b67b-3a7975331cb4\" (UID: \"825b7242-dd7e-420a-b67b-3a7975331cb4\") " Sep 18 21:20:31.946636 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.946455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-data" (OuterVolumeSpecName: "istio-data") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:20:31.949976 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.947133 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825b7242-dd7e-420a-b67b-3a7975331cb4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:20:31.960052 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.955222 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:20:31.960052 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.956172 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:20:31.960052 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.956323 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:20:31.960742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.960479 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-kube-api-access-994kx" (OuterVolumeSpecName: "kube-api-access-994kx") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "kube-api-access-994kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:20:31.960878 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.960776 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:20:31.960878 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.960785 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:20:31.961097 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:31.961077 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-token" (OuterVolumeSpecName: "istio-token") pod "825b7242-dd7e-420a-b67b-3a7975331cb4" (UID: "825b7242-dd7e-420a-b67b-3a7975331cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047359 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047405 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047419 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825b7242-dd7e-420a-b67b-3a7975331cb4-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047433 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047446 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047463 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047474 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047486 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-994kx\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-kube-api-access-994kx\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.053981 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.047499 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825b7242-dd7e-420a-b67b-3a7975331cb4-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:20:32.083271 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.083237 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" event={"ID":"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336","Type":"ContainerStarted","Data":"fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60"} Sep 18 21:20:32.084639 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.084608 2577 generic.go:356] "Generic (PLEG): container finished" podID="825b7242-dd7e-420a-b67b-3a7975331cb4" containerID="f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb" exitCode=0 Sep 18 21:20:32.084783 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.084652 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" Sep 18 21:20:32.084783 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.084699 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" event={"ID":"825b7242-dd7e-420a-b67b-3a7975331cb4","Type":"ContainerDied","Data":"f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb"} Sep 18 21:20:32.084783 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.084762 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5" event={"ID":"825b7242-dd7e-420a-b67b-3a7975331cb4","Type":"ContainerDied","Data":"a60a4afa16cee5307b01776444edd14152b4e53340f9bdac7ab62b4ea264fdf2"} Sep 18 21:20:32.084914 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.084788 2577 scope.go:117] "RemoveContainer" containerID="f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb" Sep 18 21:20:32.085087 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.085066 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:32.086902 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.086881 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:32.100053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.100031 2577 scope.go:117] "RemoveContainer" containerID="f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb" Sep 18 21:20:32.100408 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:20:32.100389 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb\": container with ID starting with f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb not found: ID does not exist" containerID="f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb" Sep 18 21:20:32.100462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.100420 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb"} err="failed to get container status \"f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb\": rpc error: code = NotFound desc = could not find container \"f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb\": container with ID starting with f163f4ae0054eb6c3bcbbdf22d14729db10f03619d3b1baa3fd16596df632ddb not found: ID does not exist" Sep 18 21:20:32.184943 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.184851 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5"] Sep 18 21:20:32.198028 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.197989 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ef01a1fd-openshift-default-fc5d6d48d-cw9l5"] Sep 18 21:20:32.818008 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.817943 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:20:32.818404 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.818187 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 21:20:32.818404 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:32.818239 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 21:20:33.362360 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:33.362326 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="825b7242-dd7e-420a-b67b-3a7975331cb4" path="/var/lib/kubelet/pods/825b7242-dd7e-420a-b67b-3a7975331cb4/volumes" Sep 18 21:20:33.818354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:33.818326 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 21:20:33.818720 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:33.818375 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 21:20:34.818657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:34.818616 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 21:20:34.819108 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:34.818684 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 21:20:35.818168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:35.818132 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 21:20:35.818353 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:35.818203 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 21:20:36.818437 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:36.818398 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 21:20:36.818810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:36.818484 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 21:20:37.822412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:37.822382 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:20:37.822810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:37.822680 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:20:37.823442 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:37.823426 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:20:48.707220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.707187 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc"] Sep 18 21:20:48.707672 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.707658 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825b7242-dd7e-420a-b67b-3a7975331cb4" containerName="istio-proxy" Sep 18 21:20:48.707672 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.707669 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="825b7242-dd7e-420a-b67b-3a7975331cb4" containerName="istio-proxy" Sep 18 21:20:48.707758 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.707745 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="825b7242-dd7e-420a-b67b-3a7975331cb4" containerName="istio-proxy" Sep 18 21:20:48.710554 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.710539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.713762 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.713737 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"openshift-service-ca.crt\"" Sep 18 21:20:48.714539 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.714521 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"kube-root-ca.crt\"" Sep 18 21:20:48.714651 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.714538 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"default-dockercfg-97mfm\"" Sep 18 21:20:48.714651 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.714635 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"simulator-w4-6ad49b-kserve-self-signed-certs\"" Sep 18 21:20:48.724819 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.724794 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc"] Sep 18 21:20:48.832012 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.831539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011ef13b-b523-4368-a2d8-1337b6c46b3c-tls-certs\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.832012 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.831686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-tmp-dir\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.832012 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.831739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-dshm\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.832012 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.831773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfqr\" (UniqueName: \"kubernetes.io/projected/011ef13b-b523-4368-a2d8-1337b6c46b3c-kube-api-access-qnfqr\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.832012 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.831829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-model-cache\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.832012 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.831874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-home\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.933608 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.933559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-dshm\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.933608 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.933623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfqr\" (UniqueName: \"kubernetes.io/projected/011ef13b-b523-4368-a2d8-1337b6c46b3c-kube-api-access-qnfqr\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.933916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.933679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-model-cache\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.933916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.933733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-home\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.933916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.933786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011ef13b-b523-4368-a2d8-1337b6c46b3c-tls-certs\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.933916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.933910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-tmp-dir\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.935168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.934380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-tmp-dir\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.935168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.934510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-model-cache\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.935168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.934621 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-home\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.937627 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.937570 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-dshm\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.939560 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.939532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011ef13b-b523-4368-a2d8-1337b6c46b3c-tls-certs\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:48.944135 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:48.944088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfqr\" (UniqueName: \"kubernetes.io/projected/011ef13b-b523-4368-a2d8-1337b6c46b3c-kube-api-access-qnfqr\") pod \"simulator-w4-6ad49b-kserve-5cf5d49875-shwtc\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:49.076864 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:49.076611 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:49.415101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:49.415072 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc"] Sep 18 21:20:49.415999 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:20:49.415950 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod011ef13b_b523_4368_a2d8_1337b6c46b3c.slice/crio-b34d790670e6b8b89e4558af18488a78781a45e99d9f477c26c2a5271be023c6 WatchSource:0}: Error finding container b34d790670e6b8b89e4558af18488a78781a45e99d9f477c26c2a5271be023c6: Status 404 returned error can't find the container with id b34d790670e6b8b89e4558af18488a78781a45e99d9f477c26c2a5271be023c6 Sep 18 21:20:50.214425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:50.214391 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" event={"ID":"011ef13b-b523-4368-a2d8-1337b6c46b3c","Type":"ContainerStarted","Data":"2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da"} Sep 18 21:20:50.214425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:50.214432 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" event={"ID":"011ef13b-b523-4368-a2d8-1337b6c46b3c","Type":"ContainerStarted","Data":"b34d790670e6b8b89e4558af18488a78781a45e99d9f477c26c2a5271be023c6"} Sep 18 21:20:50.240634 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:50.240566 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" podStartSLOduration=2.240543169 podStartE2EDuration="2.240543169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:20:50.23521938 +0000 UTC m=+879.164965255" watchObservedRunningTime="2026-09-18 21:20:50.240543169 +0000 UTC m=+879.170289043" Sep 18 21:20:56.271473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:56.271379 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerID="d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3" exitCode=0 Sep 18 21:20:56.271473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:56.271426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" event={"ID":"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98","Type":"ContainerDied","Data":"d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3"} Sep 18 21:20:56.271905 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:56.271861 2577 scope.go:117] "RemoveContainer" containerID="d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3" Sep 18 21:20:56.273158 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:56.273128 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:56.273227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:56.273198 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:56.273264 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:56.273232 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:20:57.280660 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:57.280621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" event={"ID":"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98","Type":"ContainerStarted","Data":"ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6"} Sep 18 21:20:57.791745 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:57.791693 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:20:57.793173 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:57.793031 2577 patch_prober.go:28] interesting pod/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 18 21:20:57.793173 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:57.793093 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 18 21:20:58.791550 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:58.791510 2577 patch_prober.go:28] interesting pod/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 18 21:20:58.792004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:58.791575 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 18 21:20:59.077491 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:59.077446 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:59.077730 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:59.077508 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:59.091024 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:59.090993 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:59.306231 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:59.306200 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:20:59.791840 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:59.791808 2577 patch_prober.go:28] interesting pod/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 18 21:20:59.792280 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:20:59.791859 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 18 21:21:00.791113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:00.791076 2577 patch_prober.go:28] interesting pod/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 18 21:21:00.791315 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:00.791144 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 18 21:21:01.791111 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:01.791075 2577 patch_prober.go:28] interesting pod/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 18 21:21:01.791528 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:01.791136 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 18 21:21:02.085222 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:02.085190 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:21:02.791687 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:02.791654 2577 patch_prober.go:28] interesting pod/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 18 21:21:02.792115 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:02.791707 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 18 21:21:03.795086 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:03.795054 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:21:03.796124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:03.796103 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:21:13.356558 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.356530 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:21:13.358138 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.358116 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:21:13.502461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.502429 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j"] Sep 18 21:21:13.506134 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.506109 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.508880 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.508850 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a7d035\"/\"kube-root-ca.crt\"" Sep 18 21:21:13.508880 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.508860 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a7d035\"/\"openshift-service-ca.crt\"" Sep 18 21:21:13.509715 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.509678 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a7d035\"/\"simulator-w0-a7d035-kserve-self-signed-certs\"" Sep 18 21:21:13.509838 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.509753 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a7d035\"/\"default-dockercfg-vmjmb\"" Sep 18 21:21:13.527450 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.527102 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j"] Sep 18 21:21:13.610926 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.610833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-dshm\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.610926 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.610888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tmp-dir\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.610926 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.610916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwvr\" (UniqueName: \"kubernetes.io/projected/1381edee-f95e-40f9-9c3b-94ee808cd0ba-kube-api-access-rnwvr\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.611211 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.611043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-model-cache\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.611211 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.611087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tls-certs\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.611211 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.611157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-home\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.711965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-dshm\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.712023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tmp-dir\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.712048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwvr\" (UniqueName: \"kubernetes.io/projected/1381edee-f95e-40f9-9c3b-94ee808cd0ba-kube-api-access-rnwvr\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.712151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-model-cache\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712379 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.712201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tls-certs\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712379 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.712255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-home\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712494 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.712470 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tmp-dir\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712543 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.712512 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-model-cache\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.712697 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.712677 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-home\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.714293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.714274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-dshm\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.714672 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.714650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tls-certs\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.721426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.721403 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwvr\" (UniqueName: \"kubernetes.io/projected/1381edee-f95e-40f9-9c3b-94ee808cd0ba-kube-api-access-rnwvr\") pod \"simulator-w0-a7d035-kserve-6764b854b9-t6r7j\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:13.815700 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:13.815666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:14.098250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:14.098215 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j"] Sep 18 21:21:14.103213 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:21:14.103184 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1381edee_f95e_40f9_9c3b_94ee808cd0ba.slice/crio-ff536abcb0dda5c08e107b8e850062ef16f4260fd82d65c6b0dadd13fa7024f0 WatchSource:0}: Error finding container ff536abcb0dda5c08e107b8e850062ef16f4260fd82d65c6b0dadd13fa7024f0: Status 404 returned error can't find the container with id ff536abcb0dda5c08e107b8e850062ef16f4260fd82d65c6b0dadd13fa7024f0 Sep 18 21:21:14.395594 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:14.395506 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" event={"ID":"1381edee-f95e-40f9-9c3b-94ee808cd0ba","Type":"ContainerStarted","Data":"e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5"} Sep 18 21:21:14.395594 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:14.395551 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" event={"ID":"1381edee-f95e-40f9-9c3b-94ee808cd0ba","Type":"ContainerStarted","Data":"ff536abcb0dda5c08e107b8e850062ef16f4260fd82d65c6b0dadd13fa7024f0"} Sep 18 21:21:14.437461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:14.437414 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" podStartSLOduration=1.437399477 podStartE2EDuration="1.437399477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:21:14.43439317 +0000 UTC m=+903.364139042" watchObservedRunningTime="2026-09-18 21:21:14.437399477 +0000 UTC m=+903.367145349" Sep 18 21:21:19.597366 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:19.597327 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp"] Sep 18 21:21:19.597814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:19.597591 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" containerID="cri-o://ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6" gracePeriod=30 Sep 18 21:21:20.529944 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.529907 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg"] Sep 18 21:21:20.532868 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.532845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.543413 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.543391 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1084f187-openshift-default-dockercfg-6lrtc\"" Sep 18 21:21:20.556366 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.556338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg"] Sep 18 21:21:20.692341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692296 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-istio-token\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.692831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e214317f-81dc-4555-994c-ebd2ce13088a-istiod-ca-cert\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.692831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692431 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-envoy\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.692831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdnk\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-kube-api-access-nvdnk\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.692831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-socket\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.692831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-credential-socket\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.692831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-certs\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.692831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e214317f-81dc-4555-994c-ebd2ce13088a-istio-podinfo\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.692831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.692733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-data\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.793734 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.793638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-data\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.793734 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.793704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-istio-token\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.793995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.793759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e214317f-81dc-4555-994c-ebd2ce13088a-istiod-ca-cert\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.793995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.793799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-envoy\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.793995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.793832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdnk\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-kube-api-access-nvdnk\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.793995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.793863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-socket\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.793995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.793918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-credential-socket\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.793995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.793944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-certs\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.794408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.794007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e214317f-81dc-4555-994c-ebd2ce13088a-istio-podinfo\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.794408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.794127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-data\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.794408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.794395 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-socket\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.794574 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.794457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-credential-socket\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.794574 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.794472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-certs\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.794729 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.794700 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e214317f-81dc-4555-994c-ebd2ce13088a-istiod-ca-cert\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.796478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.796440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-envoy\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.797061 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.797027 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e214317f-81dc-4555-994c-ebd2ce13088a-istio-podinfo\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.805440 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.805421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-istio-token\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.806348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.806327 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdnk\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-kube-api-access-nvdnk\") pod \"e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:20.844517 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:20.844487 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:21.161056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:21.161020 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg"] Sep 18 21:21:21.161783 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:21:21.161755 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode214317f_81dc_4555_994c_ebd2ce13088a.slice/crio-809c068ab1f47b48fbd1cff414e191b9d804d9e297922614f4f73cc5b0e5564c WatchSource:0}: Error finding container 809c068ab1f47b48fbd1cff414e191b9d804d9e297922614f4f73cc5b0e5564c: Status 404 returned error can't find the container with id 809c068ab1f47b48fbd1cff414e191b9d804d9e297922614f4f73cc5b0e5564c Sep 18 21:21:21.164274 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:21.164239 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:21:21.164352 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:21.164312 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:21:21.164352 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:21.164342 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:21:21.460075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:21.459986 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" event={"ID":"e214317f-81dc-4555-994c-ebd2ce13088a","Type":"ContainerStarted","Data":"12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072"} Sep 18 21:21:21.460075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:21.460046 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" event={"ID":"e214317f-81dc-4555-994c-ebd2ce13088a","Type":"ContainerStarted","Data":"809c068ab1f47b48fbd1cff414e191b9d804d9e297922614f4f73cc5b0e5564c"} Sep 18 21:21:21.494394 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:21.494331 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podStartSLOduration=1.494311696 podStartE2EDuration="1.494311696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:21:21.488433129 +0000 UTC m=+910.418179003" watchObservedRunningTime="2026-09-18 21:21:21.494311696 +0000 UTC m=+910.424057570" Sep 18 21:21:21.845530 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:21.845498 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:22.851485 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:22.851457 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:23.473015 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:23.472984 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:23.474066 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:23.474043 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:23.816340 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:23.816305 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:23.816579 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:23.816368 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:23.829556 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:23.829520 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:24.491302 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:24.491274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:21:24.863991 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:24.863948 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:21:25.041572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041478 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-certs\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041524 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-socket\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041547 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-data\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041573 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istiod-ca-cert\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041622 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-token\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041674 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-envoy\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041725 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsnsq\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-kube-api-access-gsnsq\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041769 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-podinfo\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041806 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-credential-socket\") pod \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\" (UID: \"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98\") " Sep 18 21:21:25.041935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041848 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:21:25.041935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.041842 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:21:25.042331 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.042071 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:21:25.042331 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.042192 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:21:25.042331 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.042285 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:21:25.042977 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.042936 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.043167 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.043150 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.043334 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.043319 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.043483 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.043468 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.043638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.043603 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.044562 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.044527 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:21:25.044665 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.044585 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:21:25.044899 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.044875 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:21:25.056126 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.056092 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-kube-api-access-gsnsq" (OuterVolumeSpecName: "kube-api-access-gsnsq") pod "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" (UID: "cb0ecd32-194b-45b8-bc8d-6a69ed55ef98"). InnerVolumeSpecName "kube-api-access-gsnsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:21:25.145297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.145259 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.145297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.145296 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnsq\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-kube-api-access-gsnsq\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.145510 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.145310 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.145510 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.145324 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:21:25.487757 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.487718 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerID="ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6" exitCode=0 Sep 18 21:21:25.487982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.487785 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" Sep 18 21:21:25.487982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.487807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" event={"ID":"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98","Type":"ContainerDied","Data":"ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6"} Sep 18 21:21:25.487982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.487878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp" event={"ID":"cb0ecd32-194b-45b8-bc8d-6a69ed55ef98","Type":"ContainerDied","Data":"c8c9716ba7f679871069b6b2c4eb18c01a3c66324c01b9a4dd5825a6f97b9680"} Sep 18 21:21:25.487982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.487909 2577 scope.go:117] "RemoveContainer" containerID="ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6" Sep 18 21:21:25.497929 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.497910 2577 scope.go:117] "RemoveContainer" containerID="d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3" Sep 18 21:21:25.507129 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.507105 2577 scope.go:117] "RemoveContainer" containerID="ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6" Sep 18 21:21:25.507496 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:21:25.507468 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6\": container with ID starting with ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6 not found: ID does not exist" containerID="ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6" Sep 18 21:21:25.507496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.507506 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6"} err="failed to get container status \"ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6\": rpc error: code = NotFound desc = could not find container \"ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6\": container with ID starting with ec59359d3049b727e1e1570d4509a30eeb54135e96cadf00eeff842e3f5d8ac6 not found: ID does not exist" Sep 18 21:21:25.507496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.507525 2577 scope.go:117] "RemoveContainer" containerID="d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3" Sep 18 21:21:25.507796 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:21:25.507778 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3\": container with ID starting with d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3 not found: ID does not exist" containerID="d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3" Sep 18 21:21:25.507892 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.507806 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3"} err="failed to get container status \"d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3\": rpc error: code = NotFound desc = could not find container \"d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3\": container with ID starting with d3825b55b4a7ac19973e375056d6eddc61ea85250d5775030df3562eaf82b1d3 not found: ID does not exist" Sep 18 21:21:25.523429 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.523399 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp"] Sep 18 21:21:25.527416 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:25.527391 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7e01a72f-gw-openshift-default-67c6864876-ffgbp"] Sep 18 21:21:27.361819 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:27.361778 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" path="/var/lib/kubelet/pods/cb0ecd32-194b-45b8-bc8d-6a69ed55ef98/volumes" Sep 18 21:21:45.628283 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:45.628245 2577 generic.go:356] "Generic (PLEG): container finished" podID="e214317f-81dc-4555-994c-ebd2ce13088a" containerID="12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072" exitCode=0 Sep 18 21:21:45.628817 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:45.628325 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" event={"ID":"e214317f-81dc-4555-994c-ebd2ce13088a","Type":"ContainerDied","Data":"12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072"} Sep 18 21:21:45.628911 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:45.628891 2577 scope.go:117] "RemoveContainer" containerID="12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072" Sep 18 21:21:45.630564 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:45.630525 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:21:45.630669 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:45.630613 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:21:45.630669 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:45.630662 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:21:46.639153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:46.639115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" event={"ID":"e214317f-81dc-4555-994c-ebd2ce13088a","Type":"ContainerStarted","Data":"0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991"} Sep 18 21:21:46.845790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:46.845243 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:46.845790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:46.845368 2577 patch_prober.go:28] interesting pod/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 18 21:21:46.845790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:46.845410 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 18 21:21:47.845082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:47.845039 2577 patch_prober.go:28] interesting pod/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 18 21:21:47.845691 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:47.845656 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 18 21:21:48.845442 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:48.845398 2577 patch_prober.go:28] interesting pod/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 18 21:21:48.845946 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:48.845480 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 18 21:21:49.845180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:49.845137 2577 patch_prober.go:28] interesting pod/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 18 21:21:49.845401 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:49.845229 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 18 21:21:50.845194 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:50.845152 2577 patch_prober.go:28] interesting pod/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 18 21:21:50.845820 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:50.845787 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 18 21:21:51.845975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:51.845907 2577 patch_prober.go:28] interesting pod/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 18 21:21:51.846489 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:51.846006 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 18 21:21:52.849508 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:52.849474 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:52.849908 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:52.849882 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:21:52.850555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:21:52.850535 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:22:06.857731 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:06.857692 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg"] Sep 18 21:22:06.858337 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:06.857913 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" containerID="cri-o://0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991" gracePeriod=30 Sep 18 21:22:07.658066 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.658015 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls"] Sep 18 21:22:07.658518 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.658505 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" Sep 18 21:22:07.658566 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.658519 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" Sep 18 21:22:07.658617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.658608 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" Sep 18 21:22:07.658653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.658621 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" Sep 18 21:22:07.660733 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.660718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.663278 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.663253 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-962c4157-openshift-default-dockercfg-9flxb\"" Sep 18 21:22:07.689073 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.689037 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls"] Sep 18 21:22:07.701439 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-podinfo\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.701620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-certs\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.701620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-socket\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.701620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-credential-socket\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.701620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-data\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.701620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-token\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.701620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850fc611-3fb1-40a1-800a-c5c39fad22dd-istiod-ca-cert\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.701919 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjw5\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-kube-api-access-cmjw5\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.701919 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.701746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-envoy\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803071 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjw5\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-kube-api-access-cmjw5\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803071 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-envoy\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803334 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-podinfo\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803334 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-certs\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-socket\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-credential-socket\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803583 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-data\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803583 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-token\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803688 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850fc611-3fb1-40a1-800a-c5c39fad22dd-istiod-ca-cert\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-certs\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803816 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-credential-socket\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803816 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-socket\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.803930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.803860 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-data\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.804362 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.804338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850fc611-3fb1-40a1-800a-c5c39fad22dd-istiod-ca-cert\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.806077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.806055 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-envoy\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.806182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.806165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-podinfo\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.824407 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.824369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-token\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.828153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.828122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjw5\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-kube-api-access-cmjw5\") pod \"e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:07.852868 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.852835 2577 patch_prober.go:28] interesting pod/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 21:22:07.853075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.852898 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 21:22:07.971044 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:07.970922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:08.254001 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:08.253933 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls"] Sep 18 21:22:08.254927 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:22:08.254902 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850fc611_3fb1_40a1_800a_c5c39fad22dd.slice/crio-607bb6857c854daa171a8c45d202bcdd96986c60da39d5d64285090b4f5ce1c0 WatchSource:0}: Error finding container 607bb6857c854daa171a8c45d202bcdd96986c60da39d5d64285090b4f5ce1c0: Status 404 returned error can't find the container with id 607bb6857c854daa171a8c45d202bcdd96986c60da39d5d64285090b4f5ce1c0 Sep 18 21:22:08.257397 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:08.257363 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:22:08.257477 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:08.257437 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:22:08.257477 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:08.257465 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:22:08.804461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:08.804414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" event={"ID":"850fc611-3fb1-40a1-800a-c5c39fad22dd","Type":"ContainerStarted","Data":"18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6"} Sep 18 21:22:08.804915 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:08.804490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" event={"ID":"850fc611-3fb1-40a1-800a-c5c39fad22dd","Type":"ContainerStarted","Data":"607bb6857c854daa171a8c45d202bcdd96986c60da39d5d64285090b4f5ce1c0"} Sep 18 21:22:08.868535 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:08.868468 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" podStartSLOduration=1.868444016 podStartE2EDuration="1.868444016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:22:08.866643955 +0000 UTC m=+957.796389829" watchObservedRunningTime="2026-09-18 21:22:08.868444016 +0000 UTC m=+957.798189890" Sep 18 21:22:08.972084 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:08.972043 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:09.977382 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:09.977343 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:10.817730 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:10.817696 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:10.818756 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:10.818732 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:12.121645 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.121578 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:22:12.149293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149259 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvdnk\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-kube-api-access-nvdnk\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149476 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149351 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-certs\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149476 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e214317f-81dc-4555-994c-ebd2ce13088a-istio-podinfo\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149582 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149478 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-istio-token\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149582 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149504 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-envoy\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149582 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149530 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-data\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149580 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e214317f-81dc-4555-994c-ebd2ce13088a-istiod-ca-cert\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149603 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-credential-socket\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149699 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-socket\") pod \"e214317f-81dc-4555-994c-ebd2ce13088a\" (UID: \"e214317f-81dc-4555-994c-ebd2ce13088a\") " Sep 18 21:22:12.149743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149720 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:12.150053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.149940 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:12.150053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.150021 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:12.150206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.150163 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:12.150318 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.150294 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.150376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.150325 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.150376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.150340 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.150376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.150353 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.150684 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.150659 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e214317f-81dc-4555-994c-ebd2ce13088a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:22:12.151759 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.151732 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e214317f-81dc-4555-994c-ebd2ce13088a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:22:12.151846 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.151756 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-kube-api-access-nvdnk" (OuterVolumeSpecName: "kube-api-access-nvdnk") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "kube-api-access-nvdnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:22:12.151846 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.151771 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:22:12.151931 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.151897 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e214317f-81dc-4555-994c-ebd2ce13088a" (UID: "e214317f-81dc-4555-994c-ebd2ce13088a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:12.251786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.251744 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvdnk\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-kube-api-access-nvdnk\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.251786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.251778 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e214317f-81dc-4555-994c-ebd2ce13088a-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.251786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.251788 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e214317f-81dc-4555-994c-ebd2ce13088a-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.251786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.251796 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e214317f-81dc-4555-994c-ebd2ce13088a-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.251786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.251805 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e214317f-81dc-4555-994c-ebd2ce13088a-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:12.831996 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.831943 2577 generic.go:356] "Generic (PLEG): container finished" podID="e214317f-81dc-4555-994c-ebd2ce13088a" containerID="0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991" exitCode=0 Sep 18 21:22:12.832209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.831990 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" event={"ID":"e214317f-81dc-4555-994c-ebd2ce13088a","Type":"ContainerDied","Data":"0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991"} Sep 18 21:22:12.832209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.832016 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" Sep 18 21:22:12.832209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.832056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg" event={"ID":"e214317f-81dc-4555-994c-ebd2ce13088a","Type":"ContainerDied","Data":"809c068ab1f47b48fbd1cff414e191b9d804d9e297922614f4f73cc5b0e5564c"} Sep 18 21:22:12.832209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.832084 2577 scope.go:117] "RemoveContainer" containerID="0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991" Sep 18 21:22:12.843830 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.843812 2577 scope.go:117] "RemoveContainer" containerID="12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072" Sep 18 21:22:12.853651 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.853631 2577 scope.go:117] "RemoveContainer" containerID="0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991" Sep 18 21:22:12.853992 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:22:12.853949 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991\": container with ID starting with 0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991 not found: ID does not exist" containerID="0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991" Sep 18 21:22:12.854051 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.854003 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991"} err="failed to get container status \"0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991\": rpc error: code = NotFound desc = could not find container \"0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991\": container with ID starting with 0974007f81d6fd592936866a5734227e5b85f62387d5eb3ed9458b703052f991 not found: ID does not exist" Sep 18 21:22:12.854051 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.854023 2577 scope.go:117] "RemoveContainer" containerID="12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072" Sep 18 21:22:12.854310 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:22:12.854290 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072\": container with ID starting with 12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072 not found: ID does not exist" containerID="12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072" Sep 18 21:22:12.854361 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.854317 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072"} err="failed to get container status \"12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072\": rpc error: code = NotFound desc = could not find container \"12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072\": container with ID starting with 12bd0cf0307505b15b6461b63b1713170b6a81968ea08f78ea0e91389c62f072 not found: ID does not exist" Sep 18 21:22:12.865202 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.865172 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg"] Sep 18 21:22:12.875067 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:12.875034 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1084f187-openshift-default-5d59cc64bb-2wbfg"] Sep 18 21:22:13.360119 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:13.360083 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" path="/var/lib/kubelet/pods/e214317f-81dc-4555-994c-ebd2ce13088a/volumes" Sep 18 21:22:39.012938 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:39.012903 2577 generic.go:356] "Generic (PLEG): container finished" podID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerID="18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6" exitCode=0 Sep 18 21:22:39.013422 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:39.012975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" event={"ID":"850fc611-3fb1-40a1-800a-c5c39fad22dd","Type":"ContainerDied","Data":"18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6"} Sep 18 21:22:39.013422 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:39.013401 2577 scope.go:117] "RemoveContainer" containerID="18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6" Sep 18 21:22:39.014525 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:39.014501 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:22:39.014592 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:39.014566 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:22:39.014633 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:39.014595 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:22:40.024559 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:40.024522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" event={"ID":"850fc611-3fb1-40a1-800a-c5c39fad22dd","Type":"ContainerStarted","Data":"fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0"} Sep 18 21:22:40.818333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:40.818297 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:40.971487 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:40.971441 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:40.972690 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:40.972658 2577 patch_prober.go:28] interesting pod/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 18 21:22:40.972842 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:40.972728 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 18 21:22:41.509084 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:41.509048 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls"] Sep 18 21:22:41.971606 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:41.971569 2577 patch_prober.go:28] interesting pod/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 18 21:22:41.971784 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:41.971636 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 18 21:22:42.046227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:42.046186 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" secret="" err="secret \"e2e-derive-962c4157-openshift-default-dockercfg-9flxb\" not found" Sep 18 21:22:42.972073 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:42.972032 2577 patch_prober.go:28] interesting pod/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 18 21:22:42.972472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:42.972095 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 18 21:22:43.056679 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:43.056634 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" containerID="cri-o://fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0" gracePeriod=30 Sep 18 21:22:51.719322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.719290 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:51.792345 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792247 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-credential-socket\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792345 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792330 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmjw5\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-kube-api-access-cmjw5\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792382 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-socket\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792423 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-certs\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792456 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-token\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792522 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-envoy\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792587 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:51.792800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792603 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-podinfo\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792671 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:51.792800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792688 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850fc611-3fb1-40a1-800a-c5c39fad22dd-istiod-ca-cert\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792722 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-data\") pod \"850fc611-3fb1-40a1-800a-c5c39fad22dd\" (UID: \"850fc611-3fb1-40a1-800a-c5c39fad22dd\") " Sep 18 21:22:51.792800 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.792761 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:51.793183 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.793057 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:51.793183 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.793100 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850fc611-3fb1-40a1-800a-c5c39fad22dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:22:51.793183 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.793173 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:51.793342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.793193 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:51.793342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.793209 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:51.793342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.793223 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:51.794948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.794918 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:22:51.795123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.794992 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:22:51.795123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.795045 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-kube-api-access-cmjw5" (OuterVolumeSpecName: "kube-api-access-cmjw5") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "kube-api-access-cmjw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:22:51.795123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.795095 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "850fc611-3fb1-40a1-800a-c5c39fad22dd" (UID: "850fc611-3fb1-40a1-800a-c5c39fad22dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:22:51.894206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.894170 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:51.894206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.894200 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:51.894206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.894209 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850fc611-3fb1-40a1-800a-c5c39fad22dd-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:51.894454 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.894220 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmjw5\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-kube-api-access-cmjw5\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:51.894454 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:51.894230 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850fc611-3fb1-40a1-800a-c5c39fad22dd-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:22:52.115634 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.115601 2577 generic.go:356] "Generic (PLEG): container finished" podID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerID="fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0" exitCode=0 Sep 18 21:22:52.115810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.115650 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" Sep 18 21:22:52.115810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.115684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" event={"ID":"850fc611-3fb1-40a1-800a-c5c39fad22dd","Type":"ContainerDied","Data":"fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0"} Sep 18 21:22:52.115810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.115738 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls" event={"ID":"850fc611-3fb1-40a1-800a-c5c39fad22dd","Type":"ContainerDied","Data":"607bb6857c854daa171a8c45d202bcdd96986c60da39d5d64285090b4f5ce1c0"} Sep 18 21:22:52.115810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.115764 2577 scope.go:117] "RemoveContainer" containerID="fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0" Sep 18 21:22:52.126212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.126192 2577 scope.go:117] "RemoveContainer" containerID="18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6" Sep 18 21:22:52.135408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.135317 2577 scope.go:117] "RemoveContainer" containerID="fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0" Sep 18 21:22:52.135682 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:22:52.135657 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0\": container with ID starting with fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0 not found: ID does not exist" containerID="fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0" Sep 18 21:22:52.135777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.135692 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0"} err="failed to get container status \"fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0\": rpc error: code = NotFound desc = could not find container \"fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0\": container with ID starting with fb6dd901d2bd99b68a2cc345dfe5c8aa98eb5386796cf574fe0e65ffd78a86f0 not found: ID does not exist" Sep 18 21:22:52.135777 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.135711 2577 scope.go:117] "RemoveContainer" containerID="18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6" Sep 18 21:22:52.136055 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:22:52.136033 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6\": container with ID starting with 18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6 not found: ID does not exist" containerID="18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6" Sep 18 21:22:52.136145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.136064 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6"} err="failed to get container status \"18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6\": rpc error: code = NotFound desc = could not find container \"18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6\": container with ID starting with 18098a6bd299ea15b902aa926c3d72c096db9e3183f21083191e06bf9b0f24b6 not found: ID does not exist" Sep 18 21:22:52.140219 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.140187 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls"] Sep 18 21:22:52.143017 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:52.142995 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-962c4157-openshift-default-6fc75d666d-bc6ls"] Sep 18 21:22:53.360851 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:22:53.360812 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" path="/var/lib/kubelet/pods/850fc611-3fb1-40a1-800a-c5c39fad22dd/volumes" Sep 18 21:23:14.828587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:14.828550 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j"] Sep 18 21:23:18.689852 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:18.689812 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb"] Sep 18 21:23:18.690354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:18.690088 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" containerID="cri-o://42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f" gracePeriod=30 Sep 18 21:23:23.955209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:23.955137 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:23:24.024212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024174 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-credential-socket\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024215 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-socket\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024452 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024253 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnx48\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-kube-api-access-bnx48\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024452 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024301 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-data\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024452 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024334 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cd998d-fbdc-4217-a92b-0abeb6151e24-istiod-ca-cert\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024452 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024389 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-envoy\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024452 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-podinfo\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024708 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024456 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-token\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024708 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024528 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-certs\") pod \"18cd998d-fbdc-4217-a92b-0abeb6151e24\" (UID: \"18cd998d-fbdc-4217-a92b-0abeb6151e24\") " Sep 18 21:23:24.024867 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024501 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:24.024867 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024850 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cd998d-fbdc-4217-a92b-0abeb6151e24-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:23:24.025046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024767 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-data" (OuterVolumeSpecName: "istio-data") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:24.025046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024814 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:24.025046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.024552 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:24.025046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.025026 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.025046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.025047 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cd998d-fbdc-4217-a92b-0abeb6151e24-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.025243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.025062 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.025243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.025076 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.025243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.025090 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.026841 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.026809 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:23:24.026939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.026838 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-kube-api-access-bnx48" (OuterVolumeSpecName: "kube-api-access-bnx48") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "kube-api-access-bnx48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:23:24.027153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.027125 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:24.027267 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.027161 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-token" (OuterVolumeSpecName: "istio-token") pod "18cd998d-fbdc-4217-a92b-0abeb6151e24" (UID: "18cd998d-fbdc-4217-a92b-0abeb6151e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:23:24.126334 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.126298 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.126515 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.126343 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.126515 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.126357 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.126515 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.126369 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnx48\" (UniqueName: \"kubernetes.io/projected/18cd998d-fbdc-4217-a92b-0abeb6151e24-kube-api-access-bnx48\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:24.330255 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.330215 2577 generic.go:356] "Generic (PLEG): container finished" podID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerID="42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f" exitCode=0 Sep 18 21:23:24.330470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.330267 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" Sep 18 21:23:24.330470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.330309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" event={"ID":"18cd998d-fbdc-4217-a92b-0abeb6151e24","Type":"ContainerDied","Data":"42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f"} Sep 18 21:23:24.330470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.330385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb" event={"ID":"18cd998d-fbdc-4217-a92b-0abeb6151e24","Type":"ContainerDied","Data":"03b00115f906eb0c5c7e6e03c96af9ca14f8933d811c88e9851e80c27346b350"} Sep 18 21:23:24.330470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.330418 2577 scope.go:117] "RemoveContainer" containerID="42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f" Sep 18 21:23:24.342439 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.342415 2577 scope.go:117] "RemoveContainer" containerID="365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7" Sep 18 21:23:24.352421 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.352399 2577 scope.go:117] "RemoveContainer" containerID="42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f" Sep 18 21:23:24.352771 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:23:24.352747 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f\": container with ID starting with 42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f not found: ID does not exist" containerID="42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f" Sep 18 21:23:24.352869 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.352785 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f"} err="failed to get container status \"42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f\": rpc error: code = NotFound desc = could not find container \"42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f\": container with ID starting with 42351051df678c5e34bfca5ce817a7ca747675450f69a9b6b94ebf5478769f0f not found: ID does not exist" Sep 18 21:23:24.352869 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.352815 2577 scope.go:117] "RemoveContainer" containerID="365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7" Sep 18 21:23:24.353154 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:23:24.353135 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7\": container with ID starting with 365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7 not found: ID does not exist" containerID="365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7" Sep 18 21:23:24.353215 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.353163 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7"} err="failed to get container status \"365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7\": rpc error: code = NotFound desc = could not find container \"365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7\": container with ID starting with 365fda7d6c6cc638d5afc50f8a92a5931b7595665878e00ef01d7f9c5959d8b7 not found: ID does not exist" Sep 18 21:23:24.357923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.357896 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb"] Sep 18 21:23:24.362503 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:24.362476 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-21df59-openshift-default-7775879dcb-8q4vb"] Sep 18 21:23:25.359864 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:25.359831 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" path="/var/lib/kubelet/pods/18cd998d-fbdc-4217-a92b-0abeb6151e24/volumes" Sep 18 21:23:29.848364 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:29.848324 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" podUID="1381edee-f95e-40f9-9c3b-94ee808cd0ba" containerName="main" containerID="cri-o://e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5" gracePeriod=45 Sep 18 21:23:30.105678 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.105612 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:23:30.192841 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.192807 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-dshm\") pod \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " Sep 18 21:23:30.193016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.192878 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-model-cache\") pod \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " Sep 18 21:23:30.193016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.192903 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tls-certs\") pod \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " Sep 18 21:23:30.193016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.192978 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnwvr\" (UniqueName: \"kubernetes.io/projected/1381edee-f95e-40f9-9c3b-94ee808cd0ba-kube-api-access-rnwvr\") pod \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " Sep 18 21:23:30.193225 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.193204 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "1381edee-f95e-40f9-9c3b-94ee808cd0ba" (UID: "1381edee-f95e-40f9-9c3b-94ee808cd0ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:30.193293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.193209 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tmp-dir\") pod \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " Sep 18 21:23:30.193347 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.193312 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-home\") pod \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\" (UID: \"1381edee-f95e-40f9-9c3b-94ee808cd0ba\") " Sep 18 21:23:30.194048 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.193420 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1381edee-f95e-40f9-9c3b-94ee808cd0ba" (UID: "1381edee-f95e-40f9-9c3b-94ee808cd0ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:30.194048 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.193603 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-home" (OuterVolumeSpecName: "home") pod "1381edee-f95e-40f9-9c3b-94ee808cd0ba" (UID: "1381edee-f95e-40f9-9c3b-94ee808cd0ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:30.194048 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.193912 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:30.194048 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.193965 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:30.194048 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.193982 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:30.196034 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.195673 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1381edee-f95e-40f9-9c3b-94ee808cd0ba-kube-api-access-rnwvr" (OuterVolumeSpecName: "kube-api-access-rnwvr") pod "1381edee-f95e-40f9-9c3b-94ee808cd0ba" (UID: "1381edee-f95e-40f9-9c3b-94ee808cd0ba"). InnerVolumeSpecName "kube-api-access-rnwvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:23:30.196034 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.195795 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1381edee-f95e-40f9-9c3b-94ee808cd0ba" (UID: "1381edee-f95e-40f9-9c3b-94ee808cd0ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:23:30.196034 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.195798 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-dshm" (OuterVolumeSpecName: "dshm") pod "1381edee-f95e-40f9-9c3b-94ee808cd0ba" (UID: "1381edee-f95e-40f9-9c3b-94ee808cd0ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:23:30.295234 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.295200 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1381edee-f95e-40f9-9c3b-94ee808cd0ba-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:30.295234 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.295230 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1381edee-f95e-40f9-9c3b-94ee808cd0ba-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:30.295234 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.295242 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwvr\" (UniqueName: \"kubernetes.io/projected/1381edee-f95e-40f9-9c3b-94ee808cd0ba-kube-api-access-rnwvr\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:23:30.383150 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.383055 2577 generic.go:356] "Generic (PLEG): container finished" podID="1381edee-f95e-40f9-9c3b-94ee808cd0ba" containerID="e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5" exitCode=0 Sep 18 21:23:30.383150 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.383104 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" Sep 18 21:23:30.383150 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.383131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" event={"ID":"1381edee-f95e-40f9-9c3b-94ee808cd0ba","Type":"ContainerDied","Data":"e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5"} Sep 18 21:23:30.383418 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.383176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j" event={"ID":"1381edee-f95e-40f9-9c3b-94ee808cd0ba","Type":"ContainerDied","Data":"ff536abcb0dda5c08e107b8e850062ef16f4260fd82d65c6b0dadd13fa7024f0"} Sep 18 21:23:30.383418 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.383193 2577 scope.go:117] "RemoveContainer" containerID="e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5" Sep 18 21:23:30.392882 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.392863 2577 scope.go:117] "RemoveContainer" containerID="e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5" Sep 18 21:23:30.393174 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:23:30.393156 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5\": container with ID starting with e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5 not found: ID does not exist" containerID="e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5" Sep 18 21:23:30.393236 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.393181 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5"} err="failed to get container status \"e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5\": rpc error: code = NotFound desc = could not find container \"e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5\": container with ID starting with e550a086ee76ad0136c65cd407e0dd51218bc434313d9a4e6b4e0061b72cf6e5 not found: ID does not exist" Sep 18 21:23:30.433005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.432967 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j"] Sep 18 21:23:30.436255 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:30.436231 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a7d035/simulator-w0-a7d035-kserve-6764b854b9-t6r7j"] Sep 18 21:23:31.361014 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:23:31.360980 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1381edee-f95e-40f9-9c3b-94ee808cd0ba" path="/var/lib/kubelet/pods/1381edee-f95e-40f9-9c3b-94ee808cd0ba/volumes" Sep 18 21:24:01.350159 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350123 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz"] Sep 18 21:24:01.350588 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350572 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1381edee-f95e-40f9-9c3b-94ee808cd0ba" containerName="main" Sep 18 21:24:01.350588 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350584 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1381edee-f95e-40f9-9c3b-94ee808cd0ba" containerName="main" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350596 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350601 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350610 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350615 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350626 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350632 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0ecd32-194b-45b8-bc8d-6a69ed55ef98" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350642 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350647 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350656 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" Sep 18 21:24:01.350664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350661 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" Sep 18 21:24:01.350975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350727 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" Sep 18 21:24:01.350975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350735 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" Sep 18 21:24:01.350975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350743 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" Sep 18 21:24:01.350975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350750 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1381edee-f95e-40f9-9c3b-94ee808cd0ba" containerName="main" Sep 18 21:24:01.350975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350756 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="850fc611-3fb1-40a1-800a-c5c39fad22dd" containerName="istio-proxy" Sep 18 21:24:01.350975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.350762 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" Sep 18 21:24:01.352930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.352903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.355571 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.355548 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"premium-w4-6ad49b-kserve-self-signed-certs\"" Sep 18 21:24:01.380116 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.380074 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz"] Sep 18 21:24:01.419328 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.419287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-model-cache\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.419328 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.419336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tmp-dir\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.419561 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.419362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tls-certs\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.419561 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.419452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvnxk\" (UniqueName: \"kubernetes.io/projected/94fee80a-ffff-4828-ad4f-12bceb38c0d9-kube-api-access-vvnxk\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.419561 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.419483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-dshm\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.419561 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.419535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-home\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520144 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-dshm\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-home\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-model-cache\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tmp-dir\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tls-certs\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520509 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvnxk\" (UniqueName: \"kubernetes.io/projected/94fee80a-ffff-4828-ad4f-12bceb38c0d9-kube-api-access-vvnxk\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520662 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520641 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-home\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-model-cache\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.520724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.520708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tmp-dir\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.522609 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.522574 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-dshm\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.522896 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.522873 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tls-certs\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.531318 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.531281 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvnxk\" (UniqueName: \"kubernetes.io/projected/94fee80a-ffff-4828-ad4f-12bceb38c0d9-kube-api-access-vvnxk\") pod \"premium-w4-6ad49b-kserve-594697f6fd-ntctz\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.670070 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.669947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:01.956788 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:01.956753 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz"] Sep 18 21:24:01.958078 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:24:01.958057 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94fee80a_ffff_4828_ad4f_12bceb38c0d9.slice/crio-88871d0687b2c496539b650cdf3910720b2781dcf3f59fb16c09d78fbab2f852 WatchSource:0}: Error finding container 88871d0687b2c496539b650cdf3910720b2781dcf3f59fb16c09d78fbab2f852: Status 404 returned error can't find the container with id 88871d0687b2c496539b650cdf3910720b2781dcf3f59fb16c09d78fbab2f852 Sep 18 21:24:02.585687 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:02.585651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" event={"ID":"94fee80a-ffff-4828-ad4f-12bceb38c0d9","Type":"ContainerStarted","Data":"b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714"} Sep 18 21:24:02.585687 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:02.585692 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" event={"ID":"94fee80a-ffff-4828-ad4f-12bceb38c0d9","Type":"ContainerStarted","Data":"88871d0687b2c496539b650cdf3910720b2781dcf3f59fb16c09d78fbab2f852"} Sep 18 21:24:02.609697 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:02.609642 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" podStartSLOduration=1.609624832 podStartE2EDuration="1.609624832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:24:02.607276737 +0000 UTC m=+1071.537022611" watchObservedRunningTime="2026-09-18 21:24:02.609624832 +0000 UTC m=+1071.539370705" Sep 18 21:24:11.670621 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:11.670581 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:11.671083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:11.670664 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:11.683083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:11.683060 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:12.664214 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:12.664184 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:24:23.569026 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.568981 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn"] Sep 18 21:24:23.569700 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.569676 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" Sep 18 21:24:23.569797 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.569702 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18cd998d-fbdc-4217-a92b-0abeb6151e24" containerName="istio-proxy" Sep 18 21:24:23.569797 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.569762 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" Sep 18 21:24:23.569797 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.569771 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" Sep 18 21:24:23.569948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.569873 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e214317f-81dc-4555-994c-ebd2ce13088a" containerName="istio-proxy" Sep 18 21:24:23.572893 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.572870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.575664 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.575641 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a2b92ced-openshift-default-dockercfg-cfv9h\"" Sep 18 21:24:23.591878 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.591844 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn"] Sep 18 21:24:23.650269 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.650226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-envoy\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.650605 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.650564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64fkz\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-kube-api-access-64fkz\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.650766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.650696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-credential-socket\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.650766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.650733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f11a7c-ee9b-4e5d-b852-891983a41081-istiod-ca-cert\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.650766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.650762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-socket\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.650935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.650785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-data\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.650935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.650821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-certs\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.651093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.650941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-token\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.651093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.651017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-podinfo\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.751572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-envoy\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.751572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64fkz\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-kube-api-access-64fkz\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.751846 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-credential-socket\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.751846 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f11a7c-ee9b-4e5d-b852-891983a41081-istiod-ca-cert\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.751846 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-socket\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.751846 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-data\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.751846 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-certs\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.752161 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-token\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.752161 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.751887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-podinfo\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.752161 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.752133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-socket\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.752328 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.752208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-credential-socket\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.752475 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.752455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-certs\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.752475 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.752468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-data\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.752589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.752503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f11a7c-ee9b-4e5d-b852-891983a41081-istiod-ca-cert\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.754518 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.754492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-envoy\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.754690 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.754674 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-podinfo\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.762802 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.762781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-token\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.762944 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.762919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64fkz\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-kube-api-access-64fkz\") pod \"e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:23.883850 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:23.883746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:24.190181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:24.190155 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn"] Sep 18 21:24:24.191333 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:24:24.191306 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f11a7c_ee9b_4e5d_b852_891983a41081.slice/crio-ae519c8df7e05864192a065bc8acad6bceb1d748f1449d78fc73ece213e2bab5 WatchSource:0}: Error finding container ae519c8df7e05864192a065bc8acad6bceb1d748f1449d78fc73ece213e2bab5: Status 404 returned error can't find the container with id ae519c8df7e05864192a065bc8acad6bceb1d748f1449d78fc73ece213e2bab5 Sep 18 21:24:24.193786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:24.193755 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:24:24.193869 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:24.193826 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:24:24.193869 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:24.193852 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:24:24.730984 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:24.730917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" event={"ID":"65f11a7c-ee9b-4e5d-b852-891983a41081","Type":"ContainerStarted","Data":"de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717"} Sep 18 21:24:24.731408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:24.731001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" event={"ID":"65f11a7c-ee9b-4e5d-b852-891983a41081","Type":"ContainerStarted","Data":"ae519c8df7e05864192a065bc8acad6bceb1d748f1449d78fc73ece213e2bab5"} Sep 18 21:24:24.763362 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:24.763299 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" podStartSLOduration=1.763278838 podStartE2EDuration="1.763278838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:24:24.758016779 +0000 UTC m=+1093.687762651" watchObservedRunningTime="2026-09-18 21:24:24.763278838 +0000 UTC m=+1093.693024714" Sep 18 21:24:24.883900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:24.883858 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:25.888716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:25.888686 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:26.755985 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:26.755934 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:26.758912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:26.758884 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:24:57.797246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:57.797198 2577 patch_prober.go:28] interesting pod/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 18 21:24:57.797724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:57.797305 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 21:24:59.995681 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:59.995539 2577 generic.go:356] "Generic (PLEG): container finished" podID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerID="de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717" exitCode=0 Sep 18 21:24:59.996241 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:59.995692 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" event={"ID":"65f11a7c-ee9b-4e5d-b852-891983a41081","Type":"ContainerDied","Data":"de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717"} Sep 18 21:24:59.996310 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:59.996253 2577 scope.go:117] "RemoveContainer" containerID="de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717" Sep 18 21:24:59.997774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:59.997735 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:24:59.997891 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:59.997828 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:24:59.997891 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:24:59.997871 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:25:01.003795 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:01.003759 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" event={"ID":"65f11a7c-ee9b-4e5d-b852-891983a41081","Type":"ContainerStarted","Data":"0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f"} Sep 18 21:25:01.884721 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:01.884685 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:25:01.885057 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:01.885035 2577 patch_prober.go:28] interesting pod/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 21:25:01.885126 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:01.885084 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 21:25:02.884276 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:02.884245 2577 patch_prober.go:28] interesting pod/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 21:25:02.884681 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:02.884295 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 21:25:03.884697 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:03.884645 2577 patch_prober.go:28] interesting pod/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 21:25:03.885175 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:03.884738 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 21:25:04.884279 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:04.884243 2577 patch_prober.go:28] interesting pod/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 21:25:04.884462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:04.884314 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 21:25:05.884292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:05.884252 2577 patch_prober.go:28] interesting pod/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 21:25:05.884781 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:05.884316 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 21:25:06.887990 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:06.887933 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:25:06.888425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:06.888340 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:25:06.889151 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:06.889133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:25:29.198937 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:29.198847 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn"] Sep 18 21:25:29.199511 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:29.199150 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" containerID="cri-o://0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f" gracePeriod=30 Sep 18 21:25:34.458320 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.458296 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:25:34.515152 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515065 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-socket\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515314 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515155 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-certs\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515314 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515221 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-podinfo\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515314 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515273 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-data\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515469 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515340 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-envoy\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515469 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515350 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:25:34.515469 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515377 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64fkz\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-kube-api-access-64fkz\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515469 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515410 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-credential-socket\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515469 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515436 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f11a7c-ee9b-4e5d-b852-891983a41081-istiod-ca-cert\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515469 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515461 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:25:34.515763 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515465 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-token\") pod \"65f11a7c-ee9b-4e5d-b852-891983a41081\" (UID: \"65f11a7c-ee9b-4e5d-b852-891983a41081\") " Sep 18 21:25:34.515763 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515551 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-data" (OuterVolumeSpecName: "istio-data") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:25:34.516402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515916 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:34.516402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515944 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:34.516402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.515987 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:34.516402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.516089 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:25:34.516402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.516182 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f11a7c-ee9b-4e5d-b852-891983a41081-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:25:34.518267 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.518236 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:25:34.518367 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.518304 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:25:34.518367 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.518321 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-token" (OuterVolumeSpecName: "istio-token") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:25:34.518556 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.518526 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-kube-api-access-64fkz" (OuterVolumeSpecName: "kube-api-access-64fkz") pod "65f11a7c-ee9b-4e5d-b852-891983a41081" (UID: "65f11a7c-ee9b-4e5d-b852-891983a41081"). InnerVolumeSpecName "kube-api-access-64fkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:25:34.617323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.617275 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:34.617323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.617319 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:34.617323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.617330 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64fkz\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-kube-api-access-64fkz\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:34.617576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.617341 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f11a7c-ee9b-4e5d-b852-891983a41081-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:34.617576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.617350 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f11a7c-ee9b-4e5d-b852-891983a41081-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:34.617576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:34.617357 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f11a7c-ee9b-4e5d-b852-891983a41081-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:25:35.246680 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.246646 2577 generic.go:356] "Generic (PLEG): container finished" podID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerID="0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f" exitCode=0 Sep 18 21:25:35.246839 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.246698 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" Sep 18 21:25:35.246839 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.246728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" event={"ID":"65f11a7c-ee9b-4e5d-b852-891983a41081","Type":"ContainerDied","Data":"0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f"} Sep 18 21:25:35.246839 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.246802 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn" event={"ID":"65f11a7c-ee9b-4e5d-b852-891983a41081","Type":"ContainerDied","Data":"ae519c8df7e05864192a065bc8acad6bceb1d748f1449d78fc73ece213e2bab5"} Sep 18 21:25:35.246839 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.246827 2577 scope.go:117] "RemoveContainer" containerID="0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f" Sep 18 21:25:35.260087 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.260064 2577 scope.go:117] "RemoveContainer" containerID="de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717" Sep 18 21:25:35.269231 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.269208 2577 scope.go:117] "RemoveContainer" containerID="0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f" Sep 18 21:25:35.269548 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:25:35.269523 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f\": container with ID starting with 0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f not found: ID does not exist" containerID="0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f" Sep 18 21:25:35.269657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.269563 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f"} err="failed to get container status \"0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f\": rpc error: code = NotFound desc = could not find container \"0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f\": container with ID starting with 0ad5e406fb3250b881549ac69e26090367f1eb0056592d53935c84ebfb692d1f not found: ID does not exist" Sep 18 21:25:35.269657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.269582 2577 scope.go:117] "RemoveContainer" containerID="de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717" Sep 18 21:25:35.269857 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:25:35.269834 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717\": container with ID starting with de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717 not found: ID does not exist" containerID="de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717" Sep 18 21:25:35.269933 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.269863 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717"} err="failed to get container status \"de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717\": rpc error: code = NotFound desc = could not find container \"de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717\": container with ID starting with de6cafc8547fc3fd176f1314d945666e9e2ffffcc0c91eb86065b611f53a0717 not found: ID does not exist" Sep 18 21:25:35.287196 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.287168 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn"] Sep 18 21:25:35.293807 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.293786 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a2b92ced-openshift-default-5cc7d5d884-pwgxn"] Sep 18 21:25:35.360357 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:35.360323 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" path="/var/lib/kubelet/pods/65f11a7c-ee9b-4e5d-b852-891983a41081/volumes" Sep 18 21:25:44.663978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.660508 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh"] Sep 18 21:25:44.663978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.661580 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" Sep 18 21:25:44.663978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.661602 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" Sep 18 21:25:44.663978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.661633 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" Sep 18 21:25:44.663978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.661649 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" Sep 18 21:25:44.663978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.661894 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" Sep 18 21:25:44.663978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.661914 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f11a7c-ee9b-4e5d-b852-891983a41081" containerName="istio-proxy" Sep 18 21:25:44.666432 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.666408 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.672684 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.672662 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-1b65ac-openshift-default-dockercfg-b7ddb\"" Sep 18 21:25:44.685980 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.685941 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh"] Sep 18 21:25:44.724111 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8652\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-kube-api-access-m8652\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.724281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-socket\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.724281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.724281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-certs\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.724281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-data\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.724281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-token\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.724599 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-credential-socket\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.724599 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724378 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-podinfo\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.724599 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.724490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-envoy\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.825749 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-credential-socket\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.825749 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-podinfo\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826036 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-envoy\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826036 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825837 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8652\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-kube-api-access-m8652\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826036 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-socket\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826036 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826036 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-certs\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826036 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-data\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826036 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.825997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-token\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826401 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.826207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-credential-socket\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826401 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.826269 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-socket\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826518 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.826495 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-data\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826570 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.826525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-certs\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.826661 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.826644 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istiod-ca-cert\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.828139 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.828105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-envoy\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.828782 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.828755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-podinfo\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.835169 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.835145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-token\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.837691 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.837670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8652\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-kube-api-access-m8652\") pod \"e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:44.976901 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:44.976806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:45.255248 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:45.255224 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh"] Sep 18 21:25:45.257249 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:25:45.257222 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20cb642_c603_4d97_8e6f_0ec218d3fa0b.slice/crio-6eccad966e1608f694a83a6aa76fac48070aaf138181ed4d46ebe4a7e32ae433 WatchSource:0}: Error finding container 6eccad966e1608f694a83a6aa76fac48070aaf138181ed4d46ebe4a7e32ae433: Status 404 returned error can't find the container with id 6eccad966e1608f694a83a6aa76fac48070aaf138181ed4d46ebe4a7e32ae433 Sep 18 21:25:45.259620 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:45.259578 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:25:45.259772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:45.259668 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:25:45.259772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:45.259721 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:25:45.310099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:45.310070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" event={"ID":"c20cb642-c603-4d97-8e6f-0ec218d3fa0b","Type":"ContainerStarted","Data":"6eccad966e1608f694a83a6aa76fac48070aaf138181ed4d46ebe4a7e32ae433"} Sep 18 21:25:46.318248 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:46.318210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" event={"ID":"c20cb642-c603-4d97-8e6f-0ec218d3fa0b","Type":"ContainerStarted","Data":"af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa"} Sep 18 21:25:46.347738 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:46.347681 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" podStartSLOduration=2.347665008 podStartE2EDuration="2.347665008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:25:46.344057824 +0000 UTC m=+1175.273803697" watchObservedRunningTime="2026-09-18 21:25:46.347665008 +0000 UTC m=+1175.277410881" Sep 18 21:25:46.977344 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:46.977311 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:46.983264 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:46.983221 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:47.324723 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:47.324694 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:25:47.325759 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:25:47.325742 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:26:01.526285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.526245 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9"] Sep 18 21:26:01.529848 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.529817 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.533393 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.533358 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-f5a86c-openshift-default-dockercfg-sz8bl\"" Sep 18 21:26:01.547517 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.547484 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9"] Sep 18 21:26:01.609000 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.608940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-credential-socket\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.609343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.609318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-certs\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.609478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.609461 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-envoy\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.609660 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.609644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-podinfo\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.609820 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.609806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-socket\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.609946 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.609931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-data\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.610101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.610085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-token\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.610246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.610231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istiod-ca-cert\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.610373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.610357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtm9b\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-kube-api-access-dtm9b\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711425 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-podinfo\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711640 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-socket\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711640 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-data\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-token\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istiod-ca-cert\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtm9b\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-kube-api-access-dtm9b\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-credential-socket\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-certs\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.711935 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.711879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-envoy\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.712162 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.712004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-socket\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.712162 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.712038 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-data\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.712317 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.712300 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-credential-socket\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.712393 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.712370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-certs\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.712514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.712492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istiod-ca-cert\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.714154 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.714133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-podinfo\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.714272 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.714170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-envoy\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.720359 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.720328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-token\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.720497 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.720474 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtm9b\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-kube-api-access-dtm9b\") pod \"e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:01.842015 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:01.841974 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:02.126452 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:02.126418 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9"] Sep 18 21:26:02.127606 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:26:02.127524 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf8b0e4_9d83_49ae_8f35_74551e94dbdc.slice/crio-0f8f096c29e4de22e4b1ff49ab1c484ca5d5e4388d91850031c84c8998a2b569 WatchSource:0}: Error finding container 0f8f096c29e4de22e4b1ff49ab1c484ca5d5e4388d91850031c84c8998a2b569: Status 404 returned error can't find the container with id 0f8f096c29e4de22e4b1ff49ab1c484ca5d5e4388d91850031c84c8998a2b569 Sep 18 21:26:02.131048 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:02.131006 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:26:02.131144 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:02.131096 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:26:02.131190 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:02.131135 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:26:02.420703 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:02.420602 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" event={"ID":"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc","Type":"ContainerStarted","Data":"61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106"} Sep 18 21:26:02.420703 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:02.420667 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" event={"ID":"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc","Type":"ContainerStarted","Data":"0f8f096c29e4de22e4b1ff49ab1c484ca5d5e4388d91850031c84c8998a2b569"} Sep 18 21:26:02.444531 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:02.444473 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" podStartSLOduration=1.444456354 podStartE2EDuration="1.444456354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:26:02.442789794 +0000 UTC m=+1191.372535666" watchObservedRunningTime="2026-09-18 21:26:02.444456354 +0000 UTC m=+1191.374202226" Sep 18 21:26:02.843078 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:02.843045 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:03.847587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:03.847558 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:04.434410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:04.434376 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:04.435514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:04.435490 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:13.411004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:13.410977 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:26:13.413561 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:13.413527 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:26:23.564132 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:23.564088 2577 generic.go:356] "Generic (PLEG): container finished" podID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerID="af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa" exitCode=0 Sep 18 21:26:23.564584 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:23.564162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" event={"ID":"c20cb642-c603-4d97-8e6f-0ec218d3fa0b","Type":"ContainerDied","Data":"af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa"} Sep 18 21:26:23.564584 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:23.564546 2577 scope.go:117] "RemoveContainer" containerID="af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa" Sep 18 21:26:23.565649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:23.565620 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:26:23.565725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:23.565689 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:26:23.565725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:23.565720 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:26:24.572644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:24.572604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" event={"ID":"c20cb642-c603-4d97-8e6f-0ec218d3fa0b","Type":"ContainerStarted","Data":"0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f"} Sep 18 21:26:24.977246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:24.977162 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:26:24.977397 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:24.977358 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 18 21:26:24.977443 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:24.977405 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 18 21:26:25.977179 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:25.977143 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 18 21:26:25.977567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:25.977205 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 18 21:26:26.977241 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:26.977203 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 18 21:26:26.977884 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:26.977842 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 18 21:26:27.977270 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:27.977238 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 18 21:26:27.977642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:27.977290 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 18 21:26:28.978051 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:28.978010 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 18 21:26:28.978896 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:28.978824 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 18 21:26:30.062270 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:30.062234 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:26:30.062762 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:30.062664 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:26:30.063561 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:30.063543 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:26:50.180734 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:50.180694 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9"] Sep 18 21:26:50.181339 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:50.180925 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" podUID="8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" containerName="istio-proxy" containerID="cri-o://61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106" gracePeriod=30 Sep 18 21:26:55.448831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.448762 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:55.602081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602043 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-envoy\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602338 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602094 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-certs\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602338 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602121 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-podinfo\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602338 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602200 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtm9b\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-kube-api-access-dtm9b\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602338 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602249 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istiod-ca-cert\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602338 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602278 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-socket\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602610 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602340 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-token\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602610 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602383 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-data\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602610 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602395 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:26:55.602610 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602436 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-credential-socket\") pod \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\" (UID: \"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc\") " Sep 18 21:26:55.602802 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602683 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:26:55.602865 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602804 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:26:55.602988 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602920 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:26:55.602988 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.602928 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:26:55.603125 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.603104 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.603161 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.603132 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.603161 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.603146 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.603230 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.603159 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.603230 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.603172 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.604529 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.604495 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:26:55.604628 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.604557 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:26:55.604810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.604786 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:26:55.604903 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.604788 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-kube-api-access-dtm9b" (OuterVolumeSpecName: "kube-api-access-dtm9b") pod "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" (UID: "8bf8b0e4-9d83-49ae-8f35-74551e94dbdc"). InnerVolumeSpecName "kube-api-access-dtm9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:26:55.704496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.704412 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.704496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.704443 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.704496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.704452 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtm9b\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-kube-api-access-dtm9b\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.704496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.704461 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:26:55.794359 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.794315 2577 generic.go:356] "Generic (PLEG): container finished" podID="8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" containerID="61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106" exitCode=0 Sep 18 21:26:55.794578 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.794367 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" event={"ID":"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc","Type":"ContainerDied","Data":"61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106"} Sep 18 21:26:55.794646 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.794420 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" Sep 18 21:26:55.794706 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.794667 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9" event={"ID":"8bf8b0e4-9d83-49ae-8f35-74551e94dbdc","Type":"ContainerDied","Data":"0f8f096c29e4de22e4b1ff49ab1c484ca5d5e4388d91850031c84c8998a2b569"} Sep 18 21:26:55.794765 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.794688 2577 scope.go:117] "RemoveContainer" containerID="61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106" Sep 18 21:26:55.805440 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.805404 2577 scope.go:117] "RemoveContainer" containerID="61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106" Sep 18 21:26:55.805799 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:26:55.805771 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106\": container with ID starting with 61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106 not found: ID does not exist" containerID="61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106" Sep 18 21:26:55.805877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.805806 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106"} err="failed to get container status \"61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106\": rpc error: code = NotFound desc = could not find container \"61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106\": container with ID starting with 61b802c424d065a8de1271222b0aa0ffefe4d1adf3714f52ac2b5e9203584106 not found: ID does not exist" Sep 18 21:26:55.824324 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.824291 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9"] Sep 18 21:26:55.833137 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:55.833101 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-f5a86c-openshift-default-76d8b9d586-6cjj9"] Sep 18 21:26:57.359675 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:57.359644 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" path="/var/lib/kubelet/pods/8bf8b0e4-9d83-49ae-8f35-74551e94dbdc/volumes" Sep 18 21:26:59.325990 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.325938 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7867-6da645"] Sep 18 21:26:59.327235 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.327208 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" containerName="istio-proxy" Sep 18 21:26:59.327378 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.327365 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" containerName="istio-proxy" Sep 18 21:26:59.327625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.327612 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf8b0e4-9d83-49ae-8f35-74551e94dbdc" containerName="istio-proxy" Sep 18 21:26:59.339514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.338436 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-6da645"] Sep 18 21:26:59.339514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.338549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-6da645" Sep 18 21:26:59.344784 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.344756 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrg6b\"" Sep 18 21:26:59.445239 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.445197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bxw\" (UniqueName: \"kubernetes.io/projected/b053b014-3b1a-432f-b3ca-511ddf47be83-kube-api-access-k2bxw\") pod \"test-curl-7867-6da645\" (UID: \"b053b014-3b1a-432f-b3ca-511ddf47be83\") " pod="opendatahub/test-curl-7867-6da645" Sep 18 21:26:59.546309 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.546261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bxw\" (UniqueName: \"kubernetes.io/projected/b053b014-3b1a-432f-b3ca-511ddf47be83-kube-api-access-k2bxw\") pod \"test-curl-7867-6da645\" (UID: \"b053b014-3b1a-432f-b3ca-511ddf47be83\") " pod="opendatahub/test-curl-7867-6da645" Sep 18 21:26:59.560052 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.560024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bxw\" (UniqueName: \"kubernetes.io/projected/b053b014-3b1a-432f-b3ca-511ddf47be83-kube-api-access-k2bxw\") pod \"test-curl-7867-6da645\" (UID: \"b053b014-3b1a-432f-b3ca-511ddf47be83\") " pod="opendatahub/test-curl-7867-6da645" Sep 18 21:26:59.667934 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.667850 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-6da645" Sep 18 21:26:59.947180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.947149 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-6da645"] Sep 18 21:26:59.947909 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:26:59.947885 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb053b014_3b1a_432f_b3ca_511ddf47be83.slice/crio-d36d2cc94ff1e7bb31e767edcb7d53ecf550101dfb3966201c01427100544dda WatchSource:0}: Error finding container d36d2cc94ff1e7bb31e767edcb7d53ecf550101dfb3966201c01427100544dda: Status 404 returned error can't find the container with id d36d2cc94ff1e7bb31e767edcb7d53ecf550101dfb3966201c01427100544dda Sep 18 21:26:59.949920 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:26:59.949903 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 21:27:00.832831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:00.832787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-6da645" event={"ID":"b053b014-3b1a-432f-b3ca-511ddf47be83","Type":"ContainerStarted","Data":"d36d2cc94ff1e7bb31e767edcb7d53ecf550101dfb3966201c01427100544dda"} Sep 18 21:27:02.848406 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:02.848374 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-6da645" event={"ID":"b053b014-3b1a-432f-b3ca-511ddf47be83","Type":"ContainerStarted","Data":"48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f"} Sep 18 21:27:02.874415 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:02.874365 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7867-6da645" podStartSLOduration=2.025489952 podStartE2EDuration="3.87434816s" totalImagesPullingTime="1.848858208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:27:02.871097187 +0000 UTC m=+1251.800843059" watchObservedRunningTime="2026-09-18 21:27:02.87434816 +0000 UTC m=+1251.804094127" Sep 18 21:27:03.465564 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:03.465525 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7867-6da645"] Sep 18 21:27:03.470641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:03.470613 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7867-6da645"] Sep 18 21:27:03.854703 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:27:03.854670 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7867-6da645" container="test-curl-7867-6da645" Sep 18 21:27:04.861040 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:04.860977 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7867-6da645" podUID="b053b014-3b1a-432f-b3ca-511ddf47be83" containerName="test-curl-7867-6da645" containerID="cri-o://48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f" gracePeriod=2 Sep 18 21:27:04.863420 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:27:04.863382 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7867-6da645\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="b053b014-3b1a-432f-b3ca-511ddf47be83" pod="opendatahub/test-curl-7867-6da645" Sep 18 21:27:06.720577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.720543 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t"] Sep 18 21:27:06.721196 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.721177 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b053b014-3b1a-432f-b3ca-511ddf47be83" containerName="test-curl-7867-6da645" Sep 18 21:27:06.721301 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.721197 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b053b014-3b1a-432f-b3ca-511ddf47be83" containerName="test-curl-7867-6da645" Sep 18 21:27:06.721360 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.721315 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b053b014-3b1a-432f-b3ca-511ddf47be83" containerName="test-curl-7867-6da645" Sep 18 21:27:06.723562 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.723541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.727337 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.727316 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-67ccba3b-openshift-default-dockercfg-6frkg\"" Sep 18 21:27:06.751919 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.751888 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t"] Sep 18 21:27:06.824713 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.824675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-token\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.824986 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.824947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-data\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.825156 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.825140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-credential-socket\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.825308 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.825291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxm2b\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-kube-api-access-fxm2b\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.825473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.825459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-socket\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.825704 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.825688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-envoy\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.825832 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.825819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-certs\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.826001 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.825988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-podinfo\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.826150 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.826134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f6b379-f4d9-4990-838d-cf9df7d02170-istiod-ca-cert\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927529 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-socket\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927704 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-envoy\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927704 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-certs\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927815 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-podinfo\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927815 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f6b379-f4d9-4990-838d-cf9df7d02170-istiod-ca-cert\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927815 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-token\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-data\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-credential-socket\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxm2b\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-kube-api-access-fxm2b\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.927994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.927949 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-socket\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.928619 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.928600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f6b379-f4d9-4990-838d-cf9df7d02170-istiod-ca-cert\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.928785 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.928760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-data\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.929010 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.928971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-certs\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.929379 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.929341 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-credential-socket\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.931846 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.931802 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-envoy\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.932231 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.932185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-podinfo\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.942573 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.942547 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-token\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:06.943277 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:06.943252 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxm2b\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-kube-api-access-fxm2b\") pod \"e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:07.036726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.036686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:07.039709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.039688 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-6da645" Sep 18 21:27:07.131384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.130880 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2bxw\" (UniqueName: \"kubernetes.io/projected/b053b014-3b1a-432f-b3ca-511ddf47be83-kube-api-access-k2bxw\") pod \"b053b014-3b1a-432f-b3ca-511ddf47be83\" (UID: \"b053b014-3b1a-432f-b3ca-511ddf47be83\") " Sep 18 21:27:07.133868 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.133830 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b053b014-3b1a-432f-b3ca-511ddf47be83-kube-api-access-k2bxw" (OuterVolumeSpecName: "kube-api-access-k2bxw") pod "b053b014-3b1a-432f-b3ca-511ddf47be83" (UID: "b053b014-3b1a-432f-b3ca-511ddf47be83"). InnerVolumeSpecName "kube-api-access-k2bxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:27:07.232453 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.232406 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bxw\" (UniqueName: \"kubernetes.io/projected/b053b014-3b1a-432f-b3ca-511ddf47be83-kube-api-access-k2bxw\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:07.318979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.318929 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t"] Sep 18 21:27:07.319450 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:27:07.319425 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f6b379_f4d9_4990_838d_cf9df7d02170.slice/crio-839f93ab7b831080aa3197852e834559ae7a282167a8325acae42e4ed47ee285 WatchSource:0}: Error finding container 839f93ab7b831080aa3197852e834559ae7a282167a8325acae42e4ed47ee285: Status 404 returned error can't find the container with id 839f93ab7b831080aa3197852e834559ae7a282167a8325acae42e4ed47ee285 Sep 18 21:27:07.321716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.321685 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:07.321810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.321751 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:07.321810 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.321782 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:07.372587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.372554 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b053b014-3b1a-432f-b3ca-511ddf47be83" path="/var/lib/kubelet/pods/b053b014-3b1a-432f-b3ca-511ddf47be83/volumes" Sep 18 21:27:07.633724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.633307 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55"] Sep 18 21:27:07.633724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.633577 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" containerID="cri-o://75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28" gracePeriod=30 Sep 18 21:27:07.885075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.884995 2577 generic.go:356] "Generic (PLEG): container finished" podID="b053b014-3b1a-432f-b3ca-511ddf47be83" containerID="48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f" exitCode=137 Sep 18 21:27:07.885075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.885047 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-6da645" Sep 18 21:27:07.885559 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.885090 2577 scope.go:117] "RemoveContainer" containerID="48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f" Sep 18 21:27:07.887236 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.887208 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" event={"ID":"d4f6b379-f4d9-4990-838d-cf9df7d02170","Type":"ContainerStarted","Data":"9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734"} Sep 18 21:27:07.887360 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.887268 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" event={"ID":"d4f6b379-f4d9-4990-838d-cf9df7d02170","Type":"ContainerStarted","Data":"839f93ab7b831080aa3197852e834559ae7a282167a8325acae42e4ed47ee285"} Sep 18 21:27:07.895133 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.895110 2577 scope.go:117] "RemoveContainer" containerID="48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f" Sep 18 21:27:07.897122 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:27:07.897054 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f\": container with ID starting with 48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f not found: ID does not exist" containerID="48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f" Sep 18 21:27:07.897122 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.897089 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f"} err="failed to get container status \"48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f\": rpc error: code = NotFound desc = could not find container \"48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f\": container with ID starting with 48c6fbdfcd3527a153b44501a85939164e0c8ff3730c8a0933fed3153e78f11f not found: ID does not exist" Sep 18 21:27:07.919794 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:07.919741 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" podStartSLOduration=1.919725014 podStartE2EDuration="1.919725014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:27:07.917008137 +0000 UTC m=+1256.846754010" watchObservedRunningTime="2026-09-18 21:27:07.919725014 +0000 UTC m=+1256.849470881" Sep 18 21:27:08.040799 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:08.040765 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:08.046775 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:08.046742 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:08.895032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:08.894994 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:08.895857 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:08.895837 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:10.858827 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:10.858789 2577 patch_prober.go:28] interesting pod/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 21:27:10.859214 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:10.858864 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 21:27:12.904198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.904135 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:27:12.922333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.922301 2577 generic.go:356] "Generic (PLEG): container finished" podID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerID="75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28" exitCode=0 Sep 18 21:27:12.922445 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.922351 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" Sep 18 21:27:12.922445 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.922351 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" event={"ID":"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b","Type":"ContainerDied","Data":"75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28"} Sep 18 21:27:12.922533 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.922465 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55" event={"ID":"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b","Type":"ContainerDied","Data":"d1ff4c5a03bb316858bf35e1f078f502736204137849b3334e7a97b7293b8950"} Sep 18 21:27:12.922533 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.922490 2577 scope.go:117] "RemoveContainer" containerID="75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28" Sep 18 21:27:12.941983 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.935817 2577 scope.go:117] "RemoveContainer" containerID="e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39" Sep 18 21:27:12.951988 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.951944 2577 scope.go:117] "RemoveContainer" containerID="75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28" Sep 18 21:27:12.952260 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:27:12.952242 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28\": container with ID starting with 75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28 not found: ID does not exist" containerID="75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28" Sep 18 21:27:12.952347 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.952269 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28"} err="failed to get container status \"75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28\": rpc error: code = NotFound desc = could not find container \"75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28\": container with ID starting with 75c51fa2043ed6988449774122cf036a67daef995becde720828815fa3261e28 not found: ID does not exist" Sep 18 21:27:12.952347 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.952290 2577 scope.go:117] "RemoveContainer" containerID="e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39" Sep 18 21:27:12.952541 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:27:12.952523 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39\": container with ID starting with e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39 not found: ID does not exist" containerID="e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39" Sep 18 21:27:12.952602 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:12.952543 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39"} err="failed to get container status \"e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39\": rpc error: code = NotFound desc = could not find container \"e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39\": container with ID starting with e13af8c17f8cc054d9d7bb31c264f670d8d89b18dd7a4fa5dc965d933d9caa39 not found: ID does not exist" Sep 18 21:27:13.000472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000446 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-certs\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.000589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000510 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-socket\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.000589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000532 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-credential-socket\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.000589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000548 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-token\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.000589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000587 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-envoy\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.000782 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000618 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istiod-ca-cert\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.000782 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000653 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-data\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.000782 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000713 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-podinfo\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.000782 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000729 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:13.000782 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000763 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp85w\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-kube-api-access-lp85w\") pod \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\" (UID: \"c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b\") " Sep 18 21:27:13.001075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000879 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:13.001075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.000896 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:13.001075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.001062 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:13.001258 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.001230 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.001258 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.001248 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.001258 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.001262 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.001400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.001276 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.001400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.001364 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:27:13.002847 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.002821 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:27:13.003035 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.002887 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-kube-api-access-lp85w" (OuterVolumeSpecName: "kube-api-access-lp85w") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "kube-api-access-lp85w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:27:13.003253 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.003232 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:27:13.003325 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.003236 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" (UID: "c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:13.102322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.102290 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.102322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.102316 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp85w\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-kube-api-access-lp85w\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.102322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.102325 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.102523 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.102334 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.102523 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.102342 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:13.249661 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.249604 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55"] Sep 18 21:27:13.253130 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.253107 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-116f55-openshift-default-668f5789c8-frx55"] Sep 18 21:27:13.360441 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:13.360409 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" path="/var/lib/kubelet/pods/c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b/volumes" Sep 18 21:27:19.174630 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.174593 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv"] Sep 18 21:27:19.175054 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.174910 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" podUID="9ae2d026-ca92-4faa-a789-e3930ac681e8" containerName="payload-processing" containerID="cri-o://0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214" gracePeriod=30 Sep 18 21:27:19.455123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.455097 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:27:19.571309 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.571276 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2d026-ca92-4faa-a789-e3930ac681e8-plugins-config-volume\") pod \"9ae2d026-ca92-4faa-a789-e3930ac681e8\" (UID: \"9ae2d026-ca92-4faa-a789-e3930ac681e8\") " Sep 18 21:27:19.571512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.571435 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwmrv\" (UniqueName: \"kubernetes.io/projected/9ae2d026-ca92-4faa-a789-e3930ac681e8-kube-api-access-dwmrv\") pod \"9ae2d026-ca92-4faa-a789-e3930ac681e8\" (UID: \"9ae2d026-ca92-4faa-a789-e3930ac681e8\") " Sep 18 21:27:19.571732 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.571705 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae2d026-ca92-4faa-a789-e3930ac681e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ae2d026-ca92-4faa-a789-e3930ac681e8" (UID: "9ae2d026-ca92-4faa-a789-e3930ac681e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:27:19.571822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.571794 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2d026-ca92-4faa-a789-e3930ac681e8-plugins-config-volume\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:19.573517 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.573491 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae2d026-ca92-4faa-a789-e3930ac681e8-kube-api-access-dwmrv" (OuterVolumeSpecName: "kube-api-access-dwmrv") pod "9ae2d026-ca92-4faa-a789-e3930ac681e8" (UID: "9ae2d026-ca92-4faa-a789-e3930ac681e8"). InnerVolumeSpecName "kube-api-access-dwmrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:27:19.673296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.673252 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmrv\" (UniqueName: \"kubernetes.io/projected/9ae2d026-ca92-4faa-a789-e3930ac681e8-kube-api-access-dwmrv\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:19.970011 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.969975 2577 generic.go:356] "Generic (PLEG): container finished" podID="9ae2d026-ca92-4faa-a789-e3930ac681e8" containerID="0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214" exitCode=0 Sep 18 21:27:19.970195 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.970017 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" Sep 18 21:27:19.970195 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.970051 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" event={"ID":"9ae2d026-ca92-4faa-a789-e3930ac681e8","Type":"ContainerDied","Data":"0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214"} Sep 18 21:27:19.970195 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.970103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv" event={"ID":"9ae2d026-ca92-4faa-a789-e3930ac681e8","Type":"ContainerDied","Data":"649b65000b78a58da4f041a50dfe2d25e31f77656947dc2587eba256f38318ee"} Sep 18 21:27:19.970195 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.970124 2577 scope.go:117] "RemoveContainer" containerID="0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214" Sep 18 21:27:19.979432 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.979417 2577 scope.go:117] "RemoveContainer" containerID="0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214" Sep 18 21:27:19.979695 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:27:19.979672 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214\": container with ID starting with 0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214 not found: ID does not exist" containerID="0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214" Sep 18 21:27:19.979762 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:19.979703 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214"} err="failed to get container status \"0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214\": rpc error: code = NotFound desc = could not find container \"0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214\": container with ID starting with 0d15cf00317f45dd5292dd37e8bf0c0d4194eda0319ab442b3ff032872dbd214 not found: ID does not exist" Sep 18 21:27:20.000297 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:20.000264 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv"] Sep 18 21:27:20.003997 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:20.003964 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f86a8-6ff5844cff-qxqmv"] Sep 18 21:27:21.360470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:21.360433 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae2d026-ca92-4faa-a789-e3930ac681e8" path="/var/lib/kubelet/pods/9ae2d026-ca92-4faa-a789-e3930ac681e8/volumes" Sep 18 21:27:28.276967 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:28.276917 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4"] Sep 18 21:27:28.277380 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:28.277165 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" containerID="cri-o://fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60" gracePeriod=30 Sep 18 21:27:29.184354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.184315 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t"] Sep 18 21:27:29.184830 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.184815 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae2d026-ca92-4faa-a789-e3930ac681e8" containerName="payload-processing" Sep 18 21:27:29.184886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.184832 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae2d026-ca92-4faa-a789-e3930ac681e8" containerName="payload-processing" Sep 18 21:27:29.184886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.184844 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" Sep 18 21:27:29.184886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.184850 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" Sep 18 21:27:29.185033 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.184932 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" Sep 18 21:27:29.185033 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.184942 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae2d026-ca92-4faa-a789-e3930ac681e8" containerName="payload-processing" Sep 18 21:27:29.187933 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.187917 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.191677 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.191655 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cad5cf-openshift-default-dockercfg-gsfvb\"" Sep 18 21:27:29.218668 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.213710 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t"] Sep 18 21:27:29.278394 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.278351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dt6z\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-kube-api-access-9dt6z\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.279030 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.279001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-credential-socket\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.279248 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.279229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-data\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.279378 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.279363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-certs\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.279498 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.279484 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-podinfo\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.279652 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.279637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a84485-4eb8-46de-8bb0-ffd9564a575b-istiod-ca-cert\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.279787 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.279771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-socket\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.279914 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.279900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-token\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.280055 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.280040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-envoy\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.381448 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-socket\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.381663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-token\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.381663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-envoy\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.381663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt6z\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-kube-api-access-9dt6z\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.381663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-credential-socket\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.381663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-data\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.381663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-certs\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.381663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-podinfo\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.382081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a84485-4eb8-46de-8bb0-ffd9564a575b-istiod-ca-cert\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.382081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381787 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-socket\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.382081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.381997 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-certs\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.382247 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.382118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-credential-socket\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.382247 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.382185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-data\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.382433 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.382408 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a84485-4eb8-46de-8bb0-ffd9564a575b-istiod-ca-cert\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.384298 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.384271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-envoy\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.384424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.384401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-podinfo\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.394747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.394726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-token\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.403761 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.403734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt6z\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-kube-api-access-9dt6z\") pod \"e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.499642 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.499568 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:29.813876 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:27:29.813847 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a84485_4eb8_46de_8bb0_ffd9564a575b.slice/crio-89dafe0fb9bea00258e7746241171460b89c51c94a530b51aee91b4566513247 WatchSource:0}: Error finding container 89dafe0fb9bea00258e7746241171460b89c51c94a530b51aee91b4566513247: Status 404 returned error can't find the container with id 89dafe0fb9bea00258e7746241171460b89c51c94a530b51aee91b4566513247 Sep 18 21:27:29.814110 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.814087 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t"] Sep 18 21:27:29.816311 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.816284 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:29.816360 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.816344 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:29.816395 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:29.816372 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:30.050617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:30.050575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" event={"ID":"81a84485-4eb8-46de-8bb0-ffd9564a575b","Type":"ContainerStarted","Data":"ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb"} Sep 18 21:27:30.050877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:30.050860 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" event={"ID":"81a84485-4eb8-46de-8bb0-ffd9564a575b","Type":"ContainerStarted","Data":"89dafe0fb9bea00258e7746241171460b89c51c94a530b51aee91b4566513247"} Sep 18 21:27:30.075100 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:30.074995 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podStartSLOduration=1.074980352 podStartE2EDuration="1.074980352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:27:30.073832274 +0000 UTC m=+1279.003578147" watchObservedRunningTime="2026-09-18 21:27:30.074980352 +0000 UTC m=+1279.004726226" Sep 18 21:27:30.500051 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:30.499949 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:30.501298 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:30.501266 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 18 21:27:30.501420 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:30.501327 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 18 21:27:31.506567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:31.506529 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:32.064656 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:32.064619 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:32.065685 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:32.065667 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:33.538528 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.538502 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.624789 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q6zg\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-kube-api-access-2q6zg\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.624853 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-socket\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.624880 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-credential-socket\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.624983 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-token\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.625027 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-certs\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.625075 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-podinfo\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.625105 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istiod-ca-cert\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.625150 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-data\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.625193 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-envoy\") pod \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\" (UID: \"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336\") " Sep 18 21:27:33.632005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.631844 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:33.634628 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.633424 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:33.634628 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.633592 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:27:33.634628 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.633807 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:33.634628 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.634019 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-data" (OuterVolumeSpecName: "istio-data") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:33.639099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.638948 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:27:33.639099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.639043 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-kube-api-access-2q6zg" (OuterVolumeSpecName: "kube-api-access-2q6zg") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "kube-api-access-2q6zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:27:33.639318 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.639164 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:27:33.639318 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.639243 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" (UID: "5bfc82b2-9a21-43d1-94fc-7f9c40ec6336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:27:33.727498 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727454 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q6zg\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-kube-api-access-2q6zg\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:33.727498 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727496 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:33.727747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727512 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:33.727747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727524 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:33.727747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727535 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:33.727747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727546 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:33.727747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727557 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:33.727747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727570 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:33.727747 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:33.727582 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:27:34.080692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.080654 2577 generic.go:356] "Generic (PLEG): container finished" podID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerID="fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60" exitCode=0 Sep 18 21:27:34.080888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.080720 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" Sep 18 21:27:34.080888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.080739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" event={"ID":"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336","Type":"ContainerDied","Data":"fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60"} Sep 18 21:27:34.080888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.080834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4" event={"ID":"5bfc82b2-9a21-43d1-94fc-7f9c40ec6336","Type":"ContainerDied","Data":"78b4f610c9490651c820170f63dd0e9799ea0a5949ab6f3ab28d86d21800c37b"} Sep 18 21:27:34.080888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.080866 2577 scope.go:117] "RemoveContainer" containerID="fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60" Sep 18 21:27:34.082844 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.082824 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerID="9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734" exitCode=0 Sep 18 21:27:34.082974 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.082895 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" event={"ID":"d4f6b379-f4d9-4990-838d-cf9df7d02170","Type":"ContainerDied","Data":"9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734"} Sep 18 21:27:34.083541 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.083512 2577 scope.go:117] "RemoveContainer" containerID="9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734" Sep 18 21:27:34.084814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.084767 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:34.084929 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.084860 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:34.084929 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.084904 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:34.092631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.092614 2577 scope.go:117] "RemoveContainer" containerID="5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783" Sep 18 21:27:34.105472 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.105446 2577 scope.go:117] "RemoveContainer" containerID="fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60" Sep 18 21:27:34.105826 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:27:34.105805 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60\": container with ID starting with fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60 not found: ID does not exist" containerID="fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60" Sep 18 21:27:34.105910 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.105835 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60"} err="failed to get container status \"fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60\": rpc error: code = NotFound desc = could not find container \"fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60\": container with ID starting with fb4eb54a23631ee6078565064245f297651214fb97d74505fd6b95ac123fdc60 not found: ID does not exist" Sep 18 21:27:34.105910 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.105858 2577 scope.go:117] "RemoveContainer" containerID="5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783" Sep 18 21:27:34.106247 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:27:34.106224 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783\": container with ID starting with 5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783 not found: ID does not exist" containerID="5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783" Sep 18 21:27:34.106306 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.106258 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783"} err="failed to get container status \"5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783\": rpc error: code = NotFound desc = could not find container \"5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783\": container with ID starting with 5d6b0290a7e6e50edf963480d573ec43c636dcc53fe68d6e9cd83a5af726d783 not found: ID does not exist" Sep 18 21:27:34.131510 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.131233 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4"] Sep 18 21:27:34.136862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:34.136818 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1f86a8-openshift-default-6c4f6bf54d-r9jp4"] Sep 18 21:27:35.092447 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:35.092414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" event={"ID":"d4f6b379-f4d9-4990-838d-cf9df7d02170","Type":"ContainerStarted","Data":"bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e"} Sep 18 21:27:35.360294 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:35.360215 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" path="/var/lib/kubelet/pods/5bfc82b2-9a21-43d1-94fc-7f9c40ec6336/volumes" Sep 18 21:27:36.037616 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:36.037573 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:36.039150 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:36.039107 2577 patch_prober.go:28] interesting pod/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 21:27:36.039293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:36.039194 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 21:27:37.037811 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:37.037762 2577 patch_prober.go:28] interesting pod/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 21:27:37.038351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:37.037843 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 21:27:38.038210 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:38.038163 2577 patch_prober.go:28] interesting pod/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 21:27:38.038723 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:38.038240 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 21:27:38.895251 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:38.895205 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:39.038146 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:39.037677 2577 patch_prober.go:28] interesting pod/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 21:27:39.038146 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:39.037747 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 21:27:40.037754 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:40.037713 2577 patch_prober.go:28] interesting pod/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 21:27:40.038062 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:40.038030 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 21:27:41.037566 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:41.037510 2577 patch_prober.go:28] interesting pod/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 21:27:41.038101 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:41.037611 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 21:27:42.042421 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.042393 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:42.043693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.043676 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:27:42.909780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.909741 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d"] Sep 18 21:27:42.910233 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910220 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" Sep 18 21:27:42.910296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910235 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" Sep 18 21:27:42.910296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910249 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" Sep 18 21:27:42.910296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910255 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" Sep 18 21:27:42.910296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910265 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" Sep 18 21:27:42.910296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910271 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" Sep 18 21:27:42.910461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910346 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d92aaa-6964-43b3-b0e2-0d52c1e9c97b" containerName="istio-proxy" Sep 18 21:27:42.910461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910357 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" Sep 18 21:27:42.910461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.910364 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bfc82b2-9a21-43d1-94fc-7f9c40ec6336" containerName="istio-proxy" Sep 18 21:27:42.913025 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.913010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:42.915457 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.915434 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"distinct-2-w4-6ad49b-kserve-self-signed-certs\"" Sep 18 21:27:42.930776 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:42.930744 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d"] Sep 18 21:27:43.026350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.026310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-home\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.026350 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.026349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-dshm\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.026577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.026489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfnk\" (UniqueName: \"kubernetes.io/projected/632cddc2-e996-4547-aea9-f782d1fb6e00-kube-api-access-tlfnk\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.026577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.026548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-tmp-dir\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.026654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.026582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632cddc2-e996-4547-aea9-f782d1fb6e00-tls-certs\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.026654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.026612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-model-cache\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.127855 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.127802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfnk\" (UniqueName: \"kubernetes.io/projected/632cddc2-e996-4547-aea9-f782d1fb6e00-kube-api-access-tlfnk\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.128400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.127899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-tmp-dir\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.128400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.127948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632cddc2-e996-4547-aea9-f782d1fb6e00-tls-certs\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.128400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.128029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-model-cache\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.128400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.128096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-home\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.128400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.128119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-dshm\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.128400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.128382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-tmp-dir\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.128725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.128416 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-model-cache\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.128725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.128686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-home\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.130754 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.130726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632cddc2-e996-4547-aea9-f782d1fb6e00-tls-certs\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.130921 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.130899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-dshm\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.139798 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.139773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfnk\" (UniqueName: \"kubernetes.io/projected/632cddc2-e996-4547-aea9-f782d1fb6e00-kube-api-access-tlfnk\") pod \"distinct-2-w4-6ad49b-kserve-77756765bb-fn97d\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.223530 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.223437 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:43.523282 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:27:43.523249 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod632cddc2_e996_4547_aea9_f782d1fb6e00.slice/crio-76d0efccbd798cdd0383beed55070c3e16f02d40a09adfebbcf6974d03beae30 WatchSource:0}: Error finding container 76d0efccbd798cdd0383beed55070c3e16f02d40a09adfebbcf6974d03beae30: Status 404 returned error can't find the container with id 76d0efccbd798cdd0383beed55070c3e16f02d40a09adfebbcf6974d03beae30 Sep 18 21:27:43.524294 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:43.524266 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d"] Sep 18 21:27:44.154520 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:44.154486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" event={"ID":"632cddc2-e996-4547-aea9-f782d1fb6e00","Type":"ContainerStarted","Data":"c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e"} Sep 18 21:27:44.154993 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:44.154528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" event={"ID":"632cddc2-e996-4547-aea9-f782d1fb6e00","Type":"ContainerStarted","Data":"76d0efccbd798cdd0383beed55070c3e16f02d40a09adfebbcf6974d03beae30"} Sep 18 21:27:44.195355 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:44.195302 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" podStartSLOduration=2.19528542 podStartE2EDuration="2.19528542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:27:44.187749222 +0000 UTC m=+1293.117495096" watchObservedRunningTime="2026-09-18 21:27:44.19528542 +0000 UTC m=+1293.125031292" Sep 18 21:27:53.223534 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:53.223498 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:53.224183 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:53.223545 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:53.265916 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:53.265880 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:54.231552 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:54.231461 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:27:55.709774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.709741 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78"] Sep 18 21:27:55.713145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.713127 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.717637 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.717618 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-67ccba3b\"/\"default-dockercfg-7hljc\"" Sep 18 21:27:55.717945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.717924 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-67ccba3b\"/\"openshift-service-ca.crt\"" Sep 18 21:27:55.718453 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.718430 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-67ccba3b\"/\"kube-root-ca.crt\"" Sep 18 21:27:55.718574 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.718457 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-67ccba3b\"/\"e2e-lifecycle-model-67ccba3b-kserve-self-signed-certs\"" Sep 18 21:27:55.729470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.729443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78"] Sep 18 21:27:55.875270 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.875226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-dshm\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.875830 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.875801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-home\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.876152 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.876134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tmp-dir\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.876303 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.876287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zst2f\" (UniqueName: \"kubernetes.io/projected/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-kube-api-access-zst2f\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.876443 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.876428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-model-cache\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.876726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.876708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tls-certs\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.977869 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.977769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tmp-dir\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.977869 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.977818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zst2f\" (UniqueName: \"kubernetes.io/projected/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-kube-api-access-zst2f\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.978128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.977931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-model-cache\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.978128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.978064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tls-certs\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.978258 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.978154 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tmp-dir\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.978258 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.978185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-dshm\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.978258 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.978238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-home\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.978370 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.978273 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-model-cache\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.978506 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.978483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-home\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.980387 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.980359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-dshm\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.980714 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.980689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tls-certs\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:55.989034 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:55.989006 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zst2f\" (UniqueName: \"kubernetes.io/projected/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-kube-api-access-zst2f\") pod \"e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:56.023936 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:56.023900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:27:56.336532 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:56.336504 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78"] Sep 18 21:27:56.338630 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:27:56.338597 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28cdb953_421b_4a6f_ad1f_d1ab0fa415c7.slice/crio-87eb64fc2ee32ec412ca8a307bff76832ba0b4782ab84f414f96c9e13f958823 WatchSource:0}: Error finding container 87eb64fc2ee32ec412ca8a307bff76832ba0b4782ab84f414f96c9e13f958823: Status 404 returned error can't find the container with id 87eb64fc2ee32ec412ca8a307bff76832ba0b4782ab84f414f96c9e13f958823 Sep 18 21:27:57.242344 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.242235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" event={"ID":"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7","Type":"ContainerStarted","Data":"80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5"} Sep 18 21:27:57.242344 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.242288 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" event={"ID":"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7","Type":"ContainerStarted","Data":"87eb64fc2ee32ec412ca8a307bff76832ba0b4782ab84f414f96c9e13f958823"} Sep 18 21:27:57.244298 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.244266 2577 generic.go:356] "Generic (PLEG): container finished" podID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerID="ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb" exitCode=0 Sep 18 21:27:57.244421 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.244333 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" event={"ID":"81a84485-4eb8-46de-8bb0-ffd9564a575b","Type":"ContainerDied","Data":"ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb"} Sep 18 21:27:57.244723 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.244707 2577 scope.go:117] "RemoveContainer" containerID="ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb" Sep 18 21:27:57.245858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.245830 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:57.245980 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.245908 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:57.246056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.246001 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:27:57.282136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:57.282085 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" podStartSLOduration=2.282067828 podStartE2EDuration="2.282067828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:27:57.280794902 +0000 UTC m=+1306.210540776" watchObservedRunningTime="2026-09-18 21:27:57.282067828 +0000 UTC m=+1306.211813701" Sep 18 21:27:58.263241 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:58.263201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" event={"ID":"81a84485-4eb8-46de-8bb0-ffd9564a575b","Type":"ContainerStarted","Data":"785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562"} Sep 18 21:27:58.500163 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:58.500116 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:27:58.501402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:58.501371 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 18 21:27:58.501538 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:58.501427 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 18 21:27:59.500012 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:59.499982 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 18 21:27:59.500401 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:27:59.500036 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 18 21:28:00.500900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:00.500862 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 18 21:28:00.501351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:00.500929 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 18 21:28:01.500272 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:01.500235 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 18 21:28:01.500473 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:01.500294 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 18 21:28:02.065221 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:02.065184 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:28:02.501059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:02.500940 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 18 21:28:02.501059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:02.501041 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 18 21:28:02.897081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:02.897050 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl"] Sep 18 21:28:02.899797 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:02.899776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:02.903114 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:02.903095 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-fd993e-openshift-default-dockercfg-8f6vg\"" Sep 18 21:28:02.920816 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:02.920787 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl"] Sep 18 21:28:03.056652 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.056608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-data\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.057019 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.056995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-podinfo\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.057180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.057162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7trf\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-kube-api-access-m7trf\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.057329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.057312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-envoy\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.057451 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.057436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.057596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.057579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-certs\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.057750 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.057735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-socket\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.057865 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.057847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-credential-socket\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.058005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.057989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-token\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.158968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-envoy\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-certs\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-socket\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-credential-socket\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-token\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-data\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-podinfo\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7trf\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-kube-api-access-m7trf\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159796 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-data\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.159998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.159871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.160154 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.160112 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-certs\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.160154 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.160125 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-socket\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.160245 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.160189 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-credential-socket\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.161752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.161731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-envoy\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.162064 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.162044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-podinfo\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.168403 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.168381 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7trf\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-kube-api-access-m7trf\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.169365 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.169348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-token\") pod \"e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.210390 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.210361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:03.513073 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.513039 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl"] Sep 18 21:28:03.514907 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:28:03.514882 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd3ac50_cc93_4b7d_899a_6c1a23901dfa.slice/crio-271c3d3424bef9edef64b67cada30b7763a1541027d0638c9d6c99e8a0339533 WatchSource:0}: Error finding container 271c3d3424bef9edef64b67cada30b7763a1541027d0638c9d6c99e8a0339533: Status 404 returned error can't find the container with id 271c3d3424bef9edef64b67cada30b7763a1541027d0638c9d6c99e8a0339533 Sep 18 21:28:03.518069 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.518033 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:28:03.518175 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.518116 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:28:03.518175 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.518151 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:28:03.625029 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.624944 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:28:03.630638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:03.630609 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:28:04.314826 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:04.314786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" event={"ID":"afd3ac50-cc93-4b7d-899a-6c1a23901dfa","Type":"ContainerStarted","Data":"bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021"} Sep 18 21:28:04.315266 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:04.314850 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" event={"ID":"afd3ac50-cc93-4b7d-899a-6c1a23901dfa","Type":"ContainerStarted","Data":"271c3d3424bef9edef64b67cada30b7763a1541027d0638c9d6c99e8a0339533"} Sep 18 21:28:04.339362 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:04.339309 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" podStartSLOduration=2.339290751 podStartE2EDuration="2.339290751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:28:04.336565195 +0000 UTC m=+1313.266311068" watchObservedRunningTime="2026-09-18 21:28:04.339290751 +0000 UTC m=+1313.269036625" Sep 18 21:28:05.210807 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:05.210767 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:05.215923 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:05.215897 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:05.323278 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:05.323243 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:05.329603 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:05.329570 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:06.024094 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:06.024060 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:28:06.024094 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:06.024103 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:28:06.036834 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:06.036810 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:28:06.343932 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:06.343898 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:28:34.516292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:34.516204 2577 generic.go:356] "Generic (PLEG): container finished" podID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerID="bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021" exitCode=0 Sep 18 21:28:34.516680 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:34.516287 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" event={"ID":"afd3ac50-cc93-4b7d-899a-6c1a23901dfa","Type":"ContainerDied","Data":"bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021"} Sep 18 21:28:34.516758 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:34.516739 2577 scope.go:117] "RemoveContainer" containerID="bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021" Sep 18 21:28:34.517877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:34.517852 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:28:34.517950 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:34.517918 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:28:34.517950 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:34.517948 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:28:35.324027 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:35.323995 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:35.525591 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:35.525555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" event={"ID":"afd3ac50-cc93-4b7d-899a-6c1a23901dfa","Type":"ContainerStarted","Data":"14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737"} Sep 18 21:28:36.210836 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:36.210800 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:36.212181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:36.212154 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 18 21:28:36.212328 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:36.212210 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 18 21:28:37.211064 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:37.211029 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 18 21:28:37.211487 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:37.211089 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 18 21:28:38.211007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:38.210972 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 18 21:28:38.211189 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:38.211026 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 18 21:28:39.210750 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:39.210719 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 18 21:28:39.210933 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:39.210785 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 18 21:28:40.211018 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:40.210943 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 18 21:28:40.211430 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:40.211048 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 18 21:28:41.215040 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:41.215007 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:41.215438 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:41.215300 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:28:41.216076 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:28:41.216055 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:29:06.433514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.433458 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg"] Sep 18 21:29:06.436250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.436230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" Sep 18 21:29:06.440407 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.440386 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-52bhz\"" Sep 18 21:29:06.440531 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.440428 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 21:29:06.451130 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.451098 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg"] Sep 18 21:29:06.470773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.464863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b9cf9-d040-4ffd-84eb-97326f43fa77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg\" (UID: \"983b9cf9-d040-4ffd-84eb-97326f43fa77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" Sep 18 21:29:06.566516 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.566478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b9cf9-d040-4ffd-84eb-97326f43fa77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg\" (UID: \"983b9cf9-d040-4ffd-84eb-97326f43fa77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" Sep 18 21:29:06.567230 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.567210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b9cf9-d040-4ffd-84eb-97326f43fa77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg\" (UID: \"983b9cf9-d040-4ffd-84eb-97326f43fa77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" Sep 18 21:29:06.747915 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:06.747836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" Sep 18 21:29:07.065432 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:07.065400 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg"] Sep 18 21:29:07.071383 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:29:07.071356 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983b9cf9_d040_4ffd_84eb_97326f43fa77.slice/crio-fadd7b6e68426ad6e44afe170be1fd510bcf4f5554051a1ae035bf47cb6b8fe2 WatchSource:0}: Error finding container fadd7b6e68426ad6e44afe170be1fd510bcf4f5554051a1ae035bf47cb6b8fe2: Status 404 returned error can't find the container with id fadd7b6e68426ad6e44afe170be1fd510bcf4f5554051a1ae035bf47cb6b8fe2 Sep 18 21:29:07.751938 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:07.751900 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" event={"ID":"983b9cf9-d040-4ffd-84eb-97326f43fa77","Type":"ContainerStarted","Data":"fadd7b6e68426ad6e44afe170be1fd510bcf4f5554051a1ae035bf47cb6b8fe2"} Sep 18 21:29:10.775398 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:10.775307 2577 generic.go:356] "Generic (PLEG): container finished" podID="983b9cf9-d040-4ffd-84eb-97326f43fa77" containerID="ee9b046b8b0b7d922386e94aeffe0e1c7241b76d5d71d35331a8a2c2bfc9acca" exitCode=0 Sep 18 21:29:10.775767 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:10.775395 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" event={"ID":"983b9cf9-d040-4ffd-84eb-97326f43fa77","Type":"ContainerDied","Data":"ee9b046b8b0b7d922386e94aeffe0e1c7241b76d5d71d35331a8a2c2bfc9acca"} Sep 18 21:29:11.939180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:11.939154 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" Sep 18 21:29:12.025877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:12.025843 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b9cf9-d040-4ffd-84eb-97326f43fa77-maas-api-service-ca\") pod \"983b9cf9-d040-4ffd-84eb-97326f43fa77\" (UID: \"983b9cf9-d040-4ffd-84eb-97326f43fa77\") " Sep 18 21:29:12.026208 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:12.026185 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983b9cf9-d040-4ffd-84eb-97326f43fa77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "983b9cf9-d040-4ffd-84eb-97326f43fa77" (UID: "983b9cf9-d040-4ffd-84eb-97326f43fa77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:29:12.026271 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:12.026257 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/983b9cf9-d040-4ffd-84eb-97326f43fa77-maas-api-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:29:12.789255 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:12.789228 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" Sep 18 21:29:12.789427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:12.789222 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg" event={"ID":"983b9cf9-d040-4ffd-84eb-97326f43fa77","Type":"ContainerDied","Data":"fadd7b6e68426ad6e44afe170be1fd510bcf4f5554051a1ae035bf47cb6b8fe2"} Sep 18 21:29:12.789427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:12.789373 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fadd7b6e68426ad6e44afe170be1fd510bcf4f5554051a1ae035bf47cb6b8fe2" Sep 18 21:29:22.922415 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:22.922359 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg"] Sep 18 21:29:22.926873 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:22.926846 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67ccba3b-p4rtg"] Sep 18 21:29:23.360310 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:23.360273 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983b9cf9-d040-4ffd-84eb-97326f43fa77" path="/var/lib/kubelet/pods/983b9cf9-d040-4ffd-84eb-97326f43fa77/volumes" Sep 18 21:29:31.101079 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:31.101031 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78"] Sep 18 21:29:46.120892 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.120850 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" podUID="28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" containerName="main" containerID="cri-o://80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5" gracePeriod=45 Sep 18 21:29:46.375797 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.375740 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:29:46.508136 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-home\") pod \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " Sep 18 21:29:46.508338 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508291 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-dshm\") pod \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " Sep 18 21:29:46.508413 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508371 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zst2f\" (UniqueName: \"kubernetes.io/projected/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-kube-api-access-zst2f\") pod \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " Sep 18 21:29:46.508476 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508419 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tmp-dir\") pod \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " Sep 18 21:29:46.508476 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508453 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tls-certs\") pod \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " Sep 18 21:29:46.508590 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508510 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-model-cache\") pod \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\" (UID: \"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7\") " Sep 18 21:29:46.508685 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508450 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-home" (OuterVolumeSpecName: "home") pod "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" (UID: "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:29:46.508685 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508651 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" (UID: "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:29:46.508903 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508882 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" (UID: "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:29:46.508995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508910 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:29:46.508995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.508944 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:29:46.510807 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.510782 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" (UID: "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:29:46.510906 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.510790 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-kube-api-access-zst2f" (OuterVolumeSpecName: "kube-api-access-zst2f") pod "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" (UID: "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7"). InnerVolumeSpecName "kube-api-access-zst2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:29:46.510906 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.510838 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-dshm" (OuterVolumeSpecName: "dshm") pod "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" (UID: "28cdb953-421b-4a6f-ad1f-d1ab0fa415c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:29:46.610128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.610095 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:29:46.610128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.610122 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zst2f\" (UniqueName: \"kubernetes.io/projected/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-kube-api-access-zst2f\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:29:46.610128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.610133 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:29:46.610361 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:46.610142 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:29:47.021346 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.021308 2577 generic.go:356] "Generic (PLEG): container finished" podID="28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" containerID="80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5" exitCode=0 Sep 18 21:29:47.021570 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.021358 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" Sep 18 21:29:47.021570 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.021370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" event={"ID":"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7","Type":"ContainerDied","Data":"80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5"} Sep 18 21:29:47.021570 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.021405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" event={"ID":"28cdb953-421b-4a6f-ad1f-d1ab0fa415c7","Type":"ContainerDied","Data":"87eb64fc2ee32ec412ca8a307bff76832ba0b4782ab84f414f96c9e13f958823"} Sep 18 21:29:47.021570 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.021423 2577 scope.go:117] "RemoveContainer" containerID="80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5" Sep 18 21:29:47.033212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.033183 2577 scope.go:117] "RemoveContainer" containerID="80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5" Sep 18 21:29:47.033561 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:29:47.033533 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5\": container with ID starting with 80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5 not found: ID does not exist" containerID="80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5" Sep 18 21:29:47.033681 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.033566 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5"} err="failed to get container status \"80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5\": rpc error: code = NotFound desc = could not find container \"80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5\": container with ID starting with 80e836c09d8c2a247b433dfd3fd83530903824bce1e83f365674afc4f42854c5 not found: ID does not exist" Sep 18 21:29:47.052897 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.049186 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78"] Sep 18 21:29:47.057798 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.057773 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78"] Sep 18 21:29:47.360850 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:47.360813 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" path="/var/lib/kubelet/pods/28cdb953-421b-4a6f-ad1f-d1ab0fa415c7/volumes" Sep 18 21:29:51.333814 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:29:51.333769 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-67ccba3b/e2e-lifecycle-model-67ccba3b-kserve-5ddf8bf847-8sf78" podUID="28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Sep 18 21:30:00.172171 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.172060 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k"] Sep 18 21:30:00.172752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.172727 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" containerName="main" Sep 18 21:30:00.172752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.172750 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" containerName="main" Sep 18 21:30:00.172912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.172761 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983b9cf9-d040-4ffd-84eb-97326f43fa77" containerName="revoke-keys" Sep 18 21:30:00.172912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.172773 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="983b9cf9-d040-4ffd-84eb-97326f43fa77" containerName="revoke-keys" Sep 18 21:30:00.172912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.172880 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="28cdb953-421b-4a6f-ad1f-d1ab0fa415c7" containerName="main" Sep 18 21:30:00.172912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.172891 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="983b9cf9-d040-4ffd-84eb-97326f43fa77" containerName="revoke-keys" Sep 18 21:30:00.177999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.177981 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq"] Sep 18 21:30:00.178092 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.178072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" Sep 18 21:30:00.186113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.181782 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-52bhz\"" Sep 18 21:30:00.186113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.185916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" Sep 18 21:30:00.196565 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.195911 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k"] Sep 18 21:30:00.198046 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.198025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" Sep 18 21:30:00.207850 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.207813 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5"] Sep 18 21:30:00.211256 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.211236 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" Sep 18 21:30:00.214803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.214782 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq"] Sep 18 21:30:00.214912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.214892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" Sep 18 21:30:00.225220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.225192 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5"] Sep 18 21:30:00.262646 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.262624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" Sep 18 21:30:00.500116 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.500083 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq"] Sep 18 21:30:00.501078 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:30:00.501047 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb57796_24f6_48eb_a56e_3f79e1f0e0f6.slice/crio-81752550cf8d13f28a5f24a15779581571d5228b4f9252f9e8bc219226a936fa WatchSource:0}: Error finding container 81752550cf8d13f28a5f24a15779581571d5228b4f9252f9e8bc219226a936fa: Status 404 returned error can't find the container with id 81752550cf8d13f28a5f24a15779581571d5228b4f9252f9e8bc219226a936fa Sep 18 21:30:00.506080 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:30:00.506057 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914e186c_5fc3_4084_8de2_6006ef81b90f.slice/crio-223924776b38f1e694713400616c3430c456444c17dc911eab1283a69e56f6f3 WatchSource:0}: Error finding container 223924776b38f1e694713400616c3430c456444c17dc911eab1283a69e56f6f3: Status 404 returned error can't find the container with id 223924776b38f1e694713400616c3430c456444c17dc911eab1283a69e56f6f3 Sep 18 21:30:00.506174 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.506148 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k"] Sep 18 21:30:00.568122 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:00.568092 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5"] Sep 18 21:30:00.568393 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:30:00.568367 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bb8984_0aa8_4174_91c6_bbc3d8bcb127.slice/crio-09b11ba215c644ab17c8a501bf7ad6de164ea5eea11c1a373ae40ef82be01506 WatchSource:0}: Error finding container 09b11ba215c644ab17c8a501bf7ad6de164ea5eea11c1a373ae40ef82be01506: Status 404 returned error can't find the container with id 09b11ba215c644ab17c8a501bf7ad6de164ea5eea11c1a373ae40ef82be01506 Sep 18 21:30:01.115934 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.115836 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" event={"ID":"f1bb8984-0aa8-4174-91c6-bbc3d8bcb127","Type":"ContainerStarted","Data":"c460146bccbaf9de009c52266d0edfc192cef8388097a2bc7192ea93b4c5c369"} Sep 18 21:30:01.115934 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.115895 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" event={"ID":"f1bb8984-0aa8-4174-91c6-bbc3d8bcb127","Type":"ContainerStarted","Data":"09b11ba215c644ab17c8a501bf7ad6de164ea5eea11c1a373ae40ef82be01506"} Sep 18 21:30:01.117549 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.117522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" event={"ID":"4fb57796-24f6-48eb-a56e-3f79e1f0e0f6","Type":"ContainerStarted","Data":"625f89daf8f395cfa6723815b8aac282adf817b975c647406e5b69a14202ac08"} Sep 18 21:30:01.117676 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.117573 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" event={"ID":"4fb57796-24f6-48eb-a56e-3f79e1f0e0f6","Type":"ContainerStarted","Data":"81752550cf8d13f28a5f24a15779581571d5228b4f9252f9e8bc219226a936fa"} Sep 18 21:30:01.119212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.119185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" event={"ID":"914e186c-5fc3-4084-8de2-6006ef81b90f","Type":"ContainerStarted","Data":"e24a6ff3e8fb600fcdac0ce4b5807b3dab47ff222323089ceba9002c5affd2ed"} Sep 18 21:30:01.119335 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.119232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" event={"ID":"914e186c-5fc3-4084-8de2-6006ef81b90f","Type":"ContainerStarted","Data":"223924776b38f1e694713400616c3430c456444c17dc911eab1283a69e56f6f3"} Sep 18 21:30:01.135500 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.135448 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" podStartSLOduration=1.135431084 podStartE2EDuration="1.135431084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:30:01.132192183 +0000 UTC m=+1430.061938061" watchObservedRunningTime="2026-09-18 21:30:01.135431084 +0000 UTC m=+1430.065176957" Sep 18 21:30:01.149597 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.149531 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" podStartSLOduration=1.149515163 podStartE2EDuration="1.149515163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:30:01.14690947 +0000 UTC m=+1430.076655342" watchObservedRunningTime="2026-09-18 21:30:01.149515163 +0000 UTC m=+1430.079261036" Sep 18 21:30:01.168938 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:01.168877 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" podStartSLOduration=1.168860849 podStartE2EDuration="1.168860849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:30:01.166651687 +0000 UTC m=+1430.096397561" watchObservedRunningTime="2026-09-18 21:30:01.168860849 +0000 UTC m=+1430.098606777" Sep 18 21:30:02.126825 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:02.126790 2577 generic.go:356] "Generic (PLEG): container finished" podID="4fb57796-24f6-48eb-a56e-3f79e1f0e0f6" containerID="625f89daf8f395cfa6723815b8aac282adf817b975c647406e5b69a14202ac08" exitCode=0 Sep 18 21:30:02.127284 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:02.126880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" event={"ID":"4fb57796-24f6-48eb-a56e-3f79e1f0e0f6","Type":"ContainerDied","Data":"625f89daf8f395cfa6723815b8aac282adf817b975c647406e5b69a14202ac08"} Sep 18 21:30:02.128652 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:02.128624 2577 generic.go:356] "Generic (PLEG): container finished" podID="914e186c-5fc3-4084-8de2-6006ef81b90f" containerID="e24a6ff3e8fb600fcdac0ce4b5807b3dab47ff222323089ceba9002c5affd2ed" exitCode=0 Sep 18 21:30:02.128785 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:02.128706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" event={"ID":"914e186c-5fc3-4084-8de2-6006ef81b90f","Type":"ContainerDied","Data":"e24a6ff3e8fb600fcdac0ce4b5807b3dab47ff222323089ceba9002c5affd2ed"} Sep 18 21:30:02.130512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:02.130470 2577 generic.go:356] "Generic (PLEG): container finished" podID="f1bb8984-0aa8-4174-91c6-bbc3d8bcb127" containerID="c460146bccbaf9de009c52266d0edfc192cef8388097a2bc7192ea93b4c5c369" exitCode=0 Sep 18 21:30:02.130512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:02.130500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" event={"ID":"f1bb8984-0aa8-4174-91c6-bbc3d8bcb127","Type":"ContainerDied","Data":"c460146bccbaf9de009c52266d0edfc192cef8388097a2bc7192ea93b4c5c369"} Sep 18 21:30:03.330827 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:03.330804 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" Sep 18 21:30:03.337623 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:03.337596 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" Sep 18 21:30:03.338627 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:03.338609 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" Sep 18 21:30:04.145366 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.145338 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" Sep 18 21:30:04.145366 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.145356 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5" event={"ID":"f1bb8984-0aa8-4174-91c6-bbc3d8bcb127","Type":"ContainerDied","Data":"09b11ba215c644ab17c8a501bf7ad6de164ea5eea11c1a373ae40ef82be01506"} Sep 18 21:30:04.145600 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.145422 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b11ba215c644ab17c8a501bf7ad6de164ea5eea11c1a373ae40ef82be01506" Sep 18 21:30:04.146858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.146828 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" Sep 18 21:30:04.146858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.146843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq" event={"ID":"4fb57796-24f6-48eb-a56e-3f79e1f0e0f6","Type":"ContainerDied","Data":"81752550cf8d13f28a5f24a15779581571d5228b4f9252f9e8bc219226a936fa"} Sep 18 21:30:04.147056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.146880 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81752550cf8d13f28a5f24a15779581571d5228b4f9252f9e8bc219226a936fa" Sep 18 21:30:04.148589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.148562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" event={"ID":"914e186c-5fc3-4084-8de2-6006ef81b90f","Type":"ContainerDied","Data":"223924776b38f1e694713400616c3430c456444c17dc911eab1283a69e56f6f3"} Sep 18 21:30:04.148705 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.148599 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k" Sep 18 21:30:04.148705 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:04.148609 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="223924776b38f1e694713400616c3430c456444c17dc911eab1283a69e56f6f3" Sep 18 21:30:16.278840 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:16.278807 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t"] Sep 18 21:30:16.279305 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:16.279036 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" containerID="cri-o://bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e" gracePeriod=30 Sep 18 21:30:21.530733 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.530702 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:30:21.590429 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590391 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f6b379-f4d9-4990-838d-cf9df7d02170-istiod-ca-cert\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.590429 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590434 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-envoy\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.590692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590482 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-socket\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.590692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590503 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-credential-socket\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.590692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590523 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxm2b\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-kube-api-access-fxm2b\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.590692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590561 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-certs\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.590692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590593 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-podinfo\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.590692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590659 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-token\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.591033 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590726 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-data\") pod \"d4f6b379-f4d9-4990-838d-cf9df7d02170\" (UID: \"d4f6b379-f4d9-4990-838d-cf9df7d02170\") " Sep 18 21:30:21.591033 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590807 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:21.591033 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.590941 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:21.591033 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.591001 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:21.591244 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.591051 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f6b379-f4d9-4990-838d-cf9df7d02170-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:30:21.591244 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.591202 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f6b379-f4d9-4990-838d-cf9df7d02170-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:21.591244 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.591209 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:21.591244 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.591221 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:21.591244 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.591235 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:21.591462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.591248 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:21.593007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.592977 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-kube-api-access-fxm2b" (OuterVolumeSpecName: "kube-api-access-fxm2b") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "kube-api-access-fxm2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:30:21.593187 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.593167 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:21.593480 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.593464 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:30:21.593529 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.593465 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4f6b379-f4d9-4990-838d-cf9df7d02170" (UID: "d4f6b379-f4d9-4990-838d-cf9df7d02170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:30:21.691851 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.691814 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxm2b\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-kube-api-access-fxm2b\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:21.691851 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.691843 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:21.691851 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.691853 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:21.691851 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.691862 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:21.692151 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:21.691871 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f6b379-f4d9-4990-838d-cf9df7d02170-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:22.276464 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.276430 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerID="bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e" exitCode=0 Sep 18 21:30:22.276649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.276478 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" Sep 18 21:30:22.276649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.276520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" event={"ID":"d4f6b379-f4d9-4990-838d-cf9df7d02170","Type":"ContainerDied","Data":"bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e"} Sep 18 21:30:22.276649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.276578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t" event={"ID":"d4f6b379-f4d9-4990-838d-cf9df7d02170","Type":"ContainerDied","Data":"839f93ab7b831080aa3197852e834559ae7a282167a8325acae42e4ed47ee285"} Sep 18 21:30:22.276649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.276600 2577 scope.go:117] "RemoveContainer" containerID="bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e" Sep 18 21:30:22.287090 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.287074 2577 scope.go:117] "RemoveContainer" containerID="9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734" Sep 18 21:30:22.296017 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.296000 2577 scope.go:117] "RemoveContainer" containerID="bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e" Sep 18 21:30:22.296273 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:30:22.296253 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e\": container with ID starting with bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e not found: ID does not exist" containerID="bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e" Sep 18 21:30:22.296321 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.296282 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e"} err="failed to get container status \"bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e\": rpc error: code = NotFound desc = could not find container \"bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e\": container with ID starting with bf680efef17dde165c49cf7dc726c19b93ed3f387691528db4b86edae22c5a6e not found: ID does not exist" Sep 18 21:30:22.296321 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.296298 2577 scope.go:117] "RemoveContainer" containerID="9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734" Sep 18 21:30:22.296554 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:30:22.296539 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734\": container with ID starting with 9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734 not found: ID does not exist" containerID="9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734" Sep 18 21:30:22.296608 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.296558 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734"} err="failed to get container status \"9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734\": rpc error: code = NotFound desc = could not find container \"9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734\": container with ID starting with 9525822228c717ade2dd4e61cebcba51e14f17cecf9b10d93c86642d23006734 not found: ID does not exist" Sep 18 21:30:22.300733 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.300712 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t"] Sep 18 21:30:22.305662 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:22.305642 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67ccba3b-openshift-default-c96c5cb88-gvz7t"] Sep 18 21:30:23.359885 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:23.359851 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" path="/var/lib/kubelet/pods/d4f6b379-f4d9-4990-838d-cf9df7d02170/volumes" Sep 18 21:30:35.206850 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.206814 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5"] Sep 18 21:30:35.207495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207464 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" Sep 18 21:30:35.207495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207492 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" Sep 18 21:30:35.207654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207511 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1bb8984-0aa8-4174-91c6-bbc3d8bcb127" containerName="cleanup" Sep 18 21:30:35.207654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207518 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1bb8984-0aa8-4174-91c6-bbc3d8bcb127" containerName="cleanup" Sep 18 21:30:35.207654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207542 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914e186c-5fc3-4084-8de2-6006ef81b90f" containerName="cleanup" Sep 18 21:30:35.207654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207549 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="914e186c-5fc3-4084-8de2-6006ef81b90f" containerName="cleanup" Sep 18 21:30:35.207654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207564 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb57796-24f6-48eb-a56e-3f79e1f0e0f6" containerName="cleanup" Sep 18 21:30:35.207654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207572 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb57796-24f6-48eb-a56e-3f79e1f0e0f6" containerName="cleanup" Sep 18 21:30:35.207875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207668 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" Sep 18 21:30:35.207875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207678 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1bb8984-0aa8-4174-91c6-bbc3d8bcb127" containerName="cleanup" Sep 18 21:30:35.207875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207685 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" Sep 18 21:30:35.207875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207691 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb57796-24f6-48eb-a56e-3f79e1f0e0f6" containerName="cleanup" Sep 18 21:30:35.207875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.207697 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="914e186c-5fc3-4084-8de2-6006ef81b90f" containerName="cleanup" Sep 18 21:30:35.210116 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.210099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" Sep 18 21:30:35.214548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.214530 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 21:30:35.214548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.214542 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-52bhz\"" Sep 18 21:30:35.236550 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.236527 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5"] Sep 18 21:30:35.335241 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.335211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36505386-13c7-4526-a069-77c60b02980d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5\" (UID: \"36505386-13c7-4526-a069-77c60b02980d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" Sep 18 21:30:35.436081 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.436037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36505386-13c7-4526-a069-77c60b02980d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5\" (UID: \"36505386-13c7-4526-a069-77c60b02980d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" Sep 18 21:30:35.436636 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.436616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36505386-13c7-4526-a069-77c60b02980d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5\" (UID: \"36505386-13c7-4526-a069-77c60b02980d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" Sep 18 21:30:35.520533 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.520396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" Sep 18 21:30:35.814402 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:35.814375 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5"] Sep 18 21:30:35.815388 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:30:35.815355 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36505386_13c7_4526_a069_77c60b02980d.slice/crio-47acfe5cc27fee22052a8f714a5ff02183f324b566eca53ee75e8525b83d4e14 WatchSource:0}: Error finding container 47acfe5cc27fee22052a8f714a5ff02183f324b566eca53ee75e8525b83d4e14: Status 404 returned error can't find the container with id 47acfe5cc27fee22052a8f714a5ff02183f324b566eca53ee75e8525b83d4e14 Sep 18 21:30:36.373228 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:36.373190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" event={"ID":"36505386-13c7-4526-a069-77c60b02980d","Type":"ContainerStarted","Data":"fbb0235fc59ce57e42db8a1d1b446c3ebe40f6d32bfeb09e30fba5792b512d37"} Sep 18 21:30:36.373615 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:36.373248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" event={"ID":"36505386-13c7-4526-a069-77c60b02980d","Type":"ContainerStarted","Data":"47acfe5cc27fee22052a8f714a5ff02183f324b566eca53ee75e8525b83d4e14"} Sep 18 21:30:36.419910 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:36.419792 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" podStartSLOduration=1.4197717619999999 podStartE2EDuration="1.419771762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:30:36.418091181 +0000 UTC m=+1465.347837057" watchObservedRunningTime="2026-09-18 21:30:36.419771762 +0000 UTC m=+1465.349517635" Sep 18 21:30:37.380705 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:37.380676 2577 generic.go:356] "Generic (PLEG): container finished" podID="36505386-13c7-4526-a069-77c60b02980d" containerID="fbb0235fc59ce57e42db8a1d1b446c3ebe40f6d32bfeb09e30fba5792b512d37" exitCode=0 Sep 18 21:30:37.381105 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:37.380728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" event={"ID":"36505386-13c7-4526-a069-77c60b02980d","Type":"ContainerDied","Data":"fbb0235fc59ce57e42db8a1d1b446c3ebe40f6d32bfeb09e30fba5792b512d37"} Sep 18 21:30:38.534725 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:38.534697 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" Sep 18 21:30:38.669766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:38.669684 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36505386-13c7-4526-a069-77c60b02980d-maas-api-service-ca\") pod \"36505386-13c7-4526-a069-77c60b02980d\" (UID: \"36505386-13c7-4526-a069-77c60b02980d\") " Sep 18 21:30:38.670120 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:38.670095 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36505386-13c7-4526-a069-77c60b02980d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36505386-13c7-4526-a069-77c60b02980d" (UID: "36505386-13c7-4526-a069-77c60b02980d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:30:38.771095 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:38.771066 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36505386-13c7-4526-a069-77c60b02980d-maas-api-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:39.393816 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:39.393787 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" Sep 18 21:30:39.393979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:39.393779 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5" event={"ID":"36505386-13c7-4526-a069-77c60b02980d","Type":"ContainerDied","Data":"47acfe5cc27fee22052a8f714a5ff02183f324b566eca53ee75e8525b83d4e14"} Sep 18 21:30:39.393979 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:39.393908 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47acfe5cc27fee22052a8f714a5ff02183f324b566eca53ee75e8525b83d4e14" Sep 18 21:30:39.972286 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:39.972234 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d"] Sep 18 21:30:39.990448 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:39.990419 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz"] Sep 18 21:30:40.004759 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:40.004714 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc"] Sep 18 21:30:50.972597 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:50.972564 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5"] Sep 18 21:30:50.975458 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:50.975434 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6ad49b-29829450-ddtt5"] Sep 18 21:30:51.359486 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:51.359443 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1bb8984-0aa8-4174-91c6-bbc3d8bcb127" path="/var/lib/kubelet/pods/f1bb8984-0aa8-4174-91c6-bbc3d8bcb127/volumes" Sep 18 21:30:51.369535 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:51.369501 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5"] Sep 18 21:30:51.389256 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:51.389222 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-6ad49b-r98f5"] Sep 18 21:30:53.360798 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:53.360758 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36505386-13c7-4526-a069-77c60b02980d" path="/var/lib/kubelet/pods/36505386-13c7-4526-a069-77c60b02980d/volumes" Sep 18 21:30:54.992240 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:54.992192 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" podUID="632cddc2-e996-4547-aea9-f782d1fb6e00" containerName="main" containerID="cri-o://c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e" gracePeriod=45 Sep 18 21:30:55.014477 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.014440 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" podUID="94fee80a-ffff-4828-ad4f-12bceb38c0d9" containerName="main" containerID="cri-o://b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714" gracePeriod=45 Sep 18 21:30:55.025349 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.025302 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" podUID="011ef13b-b523-4368-a2d8-1337b6c46b3c" containerName="main" containerID="cri-o://2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da" gracePeriod=45 Sep 18 21:30:55.292043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.292010 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:30:55.336527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.336495 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:30:55.340056 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.340032 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:30:55.357544 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357509 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-model-cache\") pod \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " Sep 18 21:30:55.357742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357588 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tmp-dir\") pod \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " Sep 18 21:30:55.357742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357634 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-home\") pod \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " Sep 18 21:30:55.357742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357667 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvnxk\" (UniqueName: \"kubernetes.io/projected/94fee80a-ffff-4828-ad4f-12bceb38c0d9-kube-api-access-vvnxk\") pod \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " Sep 18 21:30:55.357742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357718 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-dshm\") pod \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " Sep 18 21:30:55.358021 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357788 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tls-certs\") pod \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\" (UID: \"94fee80a-ffff-4828-ad4f-12bceb38c0d9\") " Sep 18 21:30:55.358021 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357837 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "94fee80a-ffff-4828-ad4f-12bceb38c0d9" (UID: "94fee80a-ffff-4828-ad4f-12bceb38c0d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.358021 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357891 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94fee80a-ffff-4828-ad4f-12bceb38c0d9" (UID: "94fee80a-ffff-4828-ad4f-12bceb38c0d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.358021 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.357993 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-home" (OuterVolumeSpecName: "home") pod "94fee80a-ffff-4828-ad4f-12bceb38c0d9" (UID: "94fee80a-ffff-4828-ad4f-12bceb38c0d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.358343 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.358322 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.358418 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.358351 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.358418 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.358365 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.360394 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.360368 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fee80a-ffff-4828-ad4f-12bceb38c0d9-kube-api-access-vvnxk" (OuterVolumeSpecName: "kube-api-access-vvnxk") pod "94fee80a-ffff-4828-ad4f-12bceb38c0d9" (UID: "94fee80a-ffff-4828-ad4f-12bceb38c0d9"). InnerVolumeSpecName "kube-api-access-vvnxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:30:55.360535 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.360505 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-dshm" (OuterVolumeSpecName: "dshm") pod "94fee80a-ffff-4828-ad4f-12bceb38c0d9" (UID: "94fee80a-ffff-4828-ad4f-12bceb38c0d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.360639 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.360613 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94fee80a-ffff-4828-ad4f-12bceb38c0d9" (UID: "94fee80a-ffff-4828-ad4f-12bceb38c0d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:30:55.459250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459217 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-dshm\") pod \"011ef13b-b523-4368-a2d8-1337b6c46b3c\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " Sep 18 21:30:55.459250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459255 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-model-cache\") pod \"632cddc2-e996-4547-aea9-f782d1fb6e00\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " Sep 18 21:30:55.459596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459283 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-tmp-dir\") pod \"632cddc2-e996-4547-aea9-f782d1fb6e00\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " Sep 18 21:30:55.459596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459377 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011ef13b-b523-4368-a2d8-1337b6c46b3c-tls-certs\") pod \"011ef13b-b523-4368-a2d8-1337b6c46b3c\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " Sep 18 21:30:55.459596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459407 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnfqr\" (UniqueName: \"kubernetes.io/projected/011ef13b-b523-4368-a2d8-1337b6c46b3c-kube-api-access-qnfqr\") pod \"011ef13b-b523-4368-a2d8-1337b6c46b3c\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " Sep 18 21:30:55.459596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459432 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-model-cache\") pod \"011ef13b-b523-4368-a2d8-1337b6c46b3c\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " Sep 18 21:30:55.459596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459458 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlfnk\" (UniqueName: \"kubernetes.io/projected/632cddc2-e996-4547-aea9-f782d1fb6e00-kube-api-access-tlfnk\") pod \"632cddc2-e996-4547-aea9-f782d1fb6e00\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " Sep 18 21:30:55.459596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459580 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "632cddc2-e996-4547-aea9-f782d1fb6e00" (UID: "632cddc2-e996-4547-aea9-f782d1fb6e00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.459994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459628 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-model-cache" (OuterVolumeSpecName: "model-cache") pod "632cddc2-e996-4547-aea9-f782d1fb6e00" (UID: "632cddc2-e996-4547-aea9-f782d1fb6e00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.459994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459681 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632cddc2-e996-4547-aea9-f782d1fb6e00-tls-certs\") pod \"632cddc2-e996-4547-aea9-f782d1fb6e00\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " Sep 18 21:30:55.459994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459789 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-dshm\") pod \"632cddc2-e996-4547-aea9-f782d1fb6e00\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " Sep 18 21:30:55.459994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459832 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "011ef13b-b523-4368-a2d8-1337b6c46b3c" (UID: "011ef13b-b523-4368-a2d8-1337b6c46b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.459994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459856 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-home\") pod \"011ef13b-b523-4368-a2d8-1337b6c46b3c\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " Sep 18 21:30:55.459994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.459980 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-home\") pod \"632cddc2-e996-4547-aea9-f782d1fb6e00\" (UID: \"632cddc2-e996-4547-aea9-f782d1fb6e00\") " Sep 18 21:30:55.460327 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460076 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-tmp-dir\") pod \"011ef13b-b523-4368-a2d8-1337b6c46b3c\" (UID: \"011ef13b-b523-4368-a2d8-1337b6c46b3c\") " Sep 18 21:30:55.460327 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460086 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-home" (OuterVolumeSpecName: "home") pod "011ef13b-b523-4368-a2d8-1337b6c46b3c" (UID: "011ef13b-b523-4368-a2d8-1337b6c46b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.460327 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460317 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-home" (OuterVolumeSpecName: "home") pod "632cddc2-e996-4547-aea9-f782d1fb6e00" (UID: "632cddc2-e996-4547-aea9-f782d1fb6e00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.460502 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460483 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "011ef13b-b523-4368-a2d8-1337b6c46b3c" (UID: "011ef13b-b523-4368-a2d8-1337b6c46b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.460657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460634 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.460728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460665 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fee80a-ffff-4828-ad4f-12bceb38c0d9-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.460728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460680 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.460728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460696 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.460728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460708 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fee80a-ffff-4828-ad4f-12bceb38c0d9-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.460728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460721 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.461005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460749 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.461005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460763 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvnxk\" (UniqueName: \"kubernetes.io/projected/94fee80a-ffff-4828-ad4f-12bceb38c0d9-kube-api-access-vvnxk\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.461005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.460776 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.462082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.462057 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-dshm" (OuterVolumeSpecName: "dshm") pod "011ef13b-b523-4368-a2d8-1337b6c46b3c" (UID: "011ef13b-b523-4368-a2d8-1337b6c46b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.462221 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.462118 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011ef13b-b523-4368-a2d8-1337b6c46b3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "011ef13b-b523-4368-a2d8-1337b6c46b3c" (UID: "011ef13b-b523-4368-a2d8-1337b6c46b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:30:55.462221 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.462136 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011ef13b-b523-4368-a2d8-1337b6c46b3c-kube-api-access-qnfqr" (OuterVolumeSpecName: "kube-api-access-qnfqr") pod "011ef13b-b523-4368-a2d8-1337b6c46b3c" (UID: "011ef13b-b523-4368-a2d8-1337b6c46b3c"). InnerVolumeSpecName "kube-api-access-qnfqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:30:55.462451 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.462432 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632cddc2-e996-4547-aea9-f782d1fb6e00-kube-api-access-tlfnk" (OuterVolumeSpecName: "kube-api-access-tlfnk") pod "632cddc2-e996-4547-aea9-f782d1fb6e00" (UID: "632cddc2-e996-4547-aea9-f782d1fb6e00"). InnerVolumeSpecName "kube-api-access-tlfnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:30:55.462517 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.462469 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-dshm" (OuterVolumeSpecName: "dshm") pod "632cddc2-e996-4547-aea9-f782d1fb6e00" (UID: "632cddc2-e996-4547-aea9-f782d1fb6e00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:30:55.462748 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.462725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/632cddc2-e996-4547-aea9-f782d1fb6e00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "632cddc2-e996-4547-aea9-f782d1fb6e00" (UID: "632cddc2-e996-4547-aea9-f782d1fb6e00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:30:55.507631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.507530 2577 generic.go:356] "Generic (PLEG): container finished" podID="632cddc2-e996-4547-aea9-f782d1fb6e00" containerID="c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e" exitCode=0 Sep 18 21:30:55.507631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.507579 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" Sep 18 21:30:55.507877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.507613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" event={"ID":"632cddc2-e996-4547-aea9-f782d1fb6e00","Type":"ContainerDied","Data":"c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e"} Sep 18 21:30:55.507877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.507668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d" event={"ID":"632cddc2-e996-4547-aea9-f782d1fb6e00","Type":"ContainerDied","Data":"76d0efccbd798cdd0383beed55070c3e16f02d40a09adfebbcf6974d03beae30"} Sep 18 21:30:55.507877 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.507691 2577 scope.go:117] "RemoveContainer" containerID="c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e" Sep 18 21:30:55.510304 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.510271 2577 generic.go:356] "Generic (PLEG): container finished" podID="94fee80a-ffff-4828-ad4f-12bceb38c0d9" containerID="b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714" exitCode=0 Sep 18 21:30:55.510462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.510305 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" Sep 18 21:30:55.510462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.510324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" event={"ID":"94fee80a-ffff-4828-ad4f-12bceb38c0d9","Type":"ContainerDied","Data":"b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714"} Sep 18 21:30:55.510462 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.510375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz" event={"ID":"94fee80a-ffff-4828-ad4f-12bceb38c0d9","Type":"ContainerDied","Data":"88871d0687b2c496539b650cdf3910720b2781dcf3f59fb16c09d78fbab2f852"} Sep 18 21:30:55.512271 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.512246 2577 generic.go:356] "Generic (PLEG): container finished" podID="011ef13b-b523-4368-a2d8-1337b6c46b3c" containerID="2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da" exitCode=0 Sep 18 21:30:55.512408 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.512297 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" Sep 18 21:30:55.512893 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.512847 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" event={"ID":"011ef13b-b523-4368-a2d8-1337b6c46b3c","Type":"ContainerDied","Data":"2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da"} Sep 18 21:30:55.513018 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.512915 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc" event={"ID":"011ef13b-b523-4368-a2d8-1337b6c46b3c","Type":"ContainerDied","Data":"b34d790670e6b8b89e4558af18488a78781a45e99d9f477c26c2a5271be023c6"} Sep 18 21:30:55.519915 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.519824 2577 scope.go:117] "RemoveContainer" containerID="c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e" Sep 18 21:30:55.523407 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:30:55.522656 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e\": container with ID starting with c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e not found: ID does not exist" containerID="c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e" Sep 18 21:30:55.523407 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.522696 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e"} err="failed to get container status \"c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e\": rpc error: code = NotFound desc = could not find container \"c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e\": container with ID starting with c0976be58ca966b8f75a5f9371dbeee05866b0857ede04097e2419c2436a848e not found: ID does not exist" Sep 18 21:30:55.523407 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.522721 2577 scope.go:117] "RemoveContainer" containerID="b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714" Sep 18 21:30:55.533838 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.533808 2577 scope.go:117] "RemoveContainer" containerID="b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714" Sep 18 21:30:55.534260 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:30:55.534234 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714\": container with ID starting with b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714 not found: ID does not exist" containerID="b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714" Sep 18 21:30:55.534358 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.534268 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714"} err="failed to get container status \"b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714\": rpc error: code = NotFound desc = could not find container \"b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714\": container with ID starting with b6eb57bfc647821aae4e6fb633a20b516decb449736368e57b20a93c4fafa714 not found: ID does not exist" Sep 18 21:30:55.534358 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.534287 2577 scope.go:117] "RemoveContainer" containerID="2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da" Sep 18 21:30:55.541766 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.541735 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz"] Sep 18 21:30:55.544488 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.544455 2577 scope.go:117] "RemoveContainer" containerID="2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da" Sep 18 21:30:55.544847 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:30:55.544823 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da\": container with ID starting with 2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da not found: ID does not exist" containerID="2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da" Sep 18 21:30:55.544903 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.544861 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da"} err="failed to get container status \"2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da\": rpc error: code = NotFound desc = could not find container \"2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da\": container with ID starting with 2ef7353bf9ed15963e7b6880b0cde2081217524ae71e623a1223d654ce8e22da not found: ID does not exist" Sep 18 21:30:55.545266 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.545249 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/premium-w4-6ad49b-kserve-594697f6fd-ntctz"] Sep 18 21:30:55.556223 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.556186 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d"] Sep 18 21:30:55.560584 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.560554 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-2-w4-6ad49b-kserve-77756765bb-fn97d"] Sep 18 21:30:55.561328 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.561307 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/011ef13b-b523-4368-a2d8-1337b6c46b3c-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.561444 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.561332 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/011ef13b-b523-4368-a2d8-1337b6c46b3c-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.561444 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.561347 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnfqr\" (UniqueName: \"kubernetes.io/projected/011ef13b-b523-4368-a2d8-1337b6c46b3c-kube-api-access-qnfqr\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.561444 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.561359 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfnk\" (UniqueName: \"kubernetes.io/projected/632cddc2-e996-4547-aea9-f782d1fb6e00-kube-api-access-tlfnk\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.561444 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.561373 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/632cddc2-e996-4547-aea9-f782d1fb6e00-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.561444 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.561384 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/632cddc2-e996-4547-aea9-f782d1fb6e00-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:30:55.574659 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.574625 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc"] Sep 18 21:30:55.578306 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:55.578279 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/simulator-w4-6ad49b-kserve-5cf5d49875-shwtc"] Sep 18 21:30:57.360089 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:57.360056 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="011ef13b-b523-4368-a2d8-1337b6c46b3c" path="/var/lib/kubelet/pods/011ef13b-b523-4368-a2d8-1337b6c46b3c/volumes" Sep 18 21:30:57.360652 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:57.360632 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="632cddc2-e996-4547-aea9-f782d1fb6e00" path="/var/lib/kubelet/pods/632cddc2-e996-4547-aea9-f782d1fb6e00/volumes" Sep 18 21:30:57.361088 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:57.361069 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94fee80a-ffff-4828-ad4f-12bceb38c0d9" path="/var/lib/kubelet/pods/94fee80a-ffff-4828-ad4f-12bceb38c0d9/volumes" Sep 18 21:30:59.427807 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:59.427767 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2"] Sep 18 21:30:59.428317 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:30:59.428075 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" containerID="cri-o://2fc3a6969a7d6730e6b9a1874812320397255ef90bf9d5f75c088da5216720f2" gracePeriod=30 Sep 18 21:31:04.583180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.583146 2577 generic.go:356] "Generic (PLEG): container finished" podID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerID="2fc3a6969a7d6730e6b9a1874812320397255ef90bf9d5f75c088da5216720f2" exitCode=0 Sep 18 21:31:04.583668 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.583216 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" event={"ID":"1ca8d284-0164-444b-bb30-01745ccdf5df","Type":"ContainerDied","Data":"2fc3a6969a7d6730e6b9a1874812320397255ef90bf9d5f75c088da5216720f2"} Sep 18 21:31:04.583668 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.583279 2577 scope.go:117] "RemoveContainer" containerID="bc0db5dbdd5ff073062ac6efd145a9dae30f61bfcaecfbb97037203ae178162c" Sep 18 21:31:04.693404 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.693348 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:31:04.855709 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855673 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-data\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.855900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855721 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm2qs\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-kube-api-access-hm2qs\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.855900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855772 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-socket\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.855900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855793 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-credential-socket\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.855900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855833 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-certs\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.855900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855860 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-podinfo\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.855900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855895 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-token\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.856250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855925 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca8d284-0164-444b-bb30-01745ccdf5df-istiod-ca-cert\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.856250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855949 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-envoy\") pod \"1ca8d284-0164-444b-bb30-01745ccdf5df\" (UID: \"1ca8d284-0164-444b-bb30-01745ccdf5df\") " Sep 18 21:31:04.856250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.855997 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:31:04.856250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.856181 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:31:04.856250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.856227 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:31:04.856496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.856339 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca8d284-0164-444b-bb30-01745ccdf5df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:31:04.856496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.856348 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:31:04.856496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.856417 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:04.856496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.856436 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:04.856496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.856451 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:04.858219 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.858191 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:31:04.858347 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.858245 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:31:04.858453 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.858433 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:31:04.858562 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.858545 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-kube-api-access-hm2qs" (OuterVolumeSpecName: "kube-api-access-hm2qs") pod "1ca8d284-0164-444b-bb30-01745ccdf5df" (UID: "1ca8d284-0164-444b-bb30-01745ccdf5df"). InnerVolumeSpecName "kube-api-access-hm2qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:31:04.957626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.957549 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm2qs\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-kube-api-access-hm2qs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:04.957626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.957583 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:04.957626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.957594 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:04.957626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.957602 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:04.957626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.957611 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca8d284-0164-444b-bb30-01745ccdf5df-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:04.957626 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:04.957618 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca8d284-0164-444b-bb30-01745ccdf5df-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:05.591813 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:05.591780 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" Sep 18 21:31:05.591813 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:05.591795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2" event={"ID":"1ca8d284-0164-444b-bb30-01745ccdf5df","Type":"ContainerDied","Data":"6b44ec2c1485b442af8df8d5162365bfdad97e1e2b634c44d6688a7131a21d59"} Sep 18 21:31:05.592356 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:05.591854 2577 scope.go:117] "RemoveContainer" containerID="2fc3a6969a7d6730e6b9a1874812320397255ef90bf9d5f75c088da5216720f2" Sep 18 21:31:05.613291 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:05.613262 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2"] Sep 18 21:31:05.623910 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:05.623850 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6ad49b-openshift-default-7d7bb455dd-vxzd2"] Sep 18 21:31:07.359646 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:07.359614 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" path="/var/lib/kubelet/pods/1ca8d284-0164-444b-bb30-01745ccdf5df/volumes" Sep 18 21:31:13.456212 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.456181 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:31:13.463790 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.463752 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:31:13.972589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.972555 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp"] Sep 18 21:31:13.973063 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973046 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973066 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f6b379-f4d9-4990-838d-cf9df7d02170" containerName="istio-proxy" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973082 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973089 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973098 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011ef13b-b523-4368-a2d8-1337b6c46b3c" containerName="main" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973104 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="011ef13b-b523-4368-a2d8-1337b6c46b3c" containerName="main" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973111 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973116 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973125 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fee80a-ffff-4828-ad4f-12bceb38c0d9" containerName="main" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973132 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fee80a-ffff-4828-ad4f-12bceb38c0d9" containerName="main" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973141 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="632cddc2-e996-4547-aea9-f782d1fb6e00" containerName="main" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973149 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="632cddc2-e996-4547-aea9-f782d1fb6e00" containerName="main" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973160 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36505386-13c7-4526-a069-77c60b02980d" containerName="revoke-keys" Sep 18 21:31:13.973182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973166 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36505386-13c7-4526-a069-77c60b02980d" containerName="revoke-keys" Sep 18 21:31:13.973890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973243 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="011ef13b-b523-4368-a2d8-1337b6c46b3c" containerName="main" Sep 18 21:31:13.973890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973252 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" Sep 18 21:31:13.973890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973258 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca8d284-0164-444b-bb30-01745ccdf5df" containerName="istio-proxy" Sep 18 21:31:13.973890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973266 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="632cddc2-e996-4547-aea9-f782d1fb6e00" containerName="main" Sep 18 21:31:13.973890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973273 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="94fee80a-ffff-4828-ad4f-12bceb38c0d9" containerName="main" Sep 18 21:31:13.973890 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.973278 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36505386-13c7-4526-a069-77c60b02980d" containerName="revoke-keys" Sep 18 21:31:13.978088 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.978066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:13.983218 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.983194 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8424a6-openshift-default-dockercfg-hxdnt\"" Sep 18 21:31:13.992170 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:13.992139 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp"] Sep 18 21:31:14.046949 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.046898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-podinfo\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.047168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.046965 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istiod-ca-cert\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.047168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.047009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q7ld\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-kube-api-access-8q7ld\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.047168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.047112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-socket\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.047168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.047157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-token\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.047399 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.047217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-envoy\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.047399 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.047251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-credential-socket\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.047399 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.047292 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-certs\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.047399 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.047368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-data\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.148913 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.148880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-envoy\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.148924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-credential-socket\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.148976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-certs\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.149064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-data\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.149103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-podinfo\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.149126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istiod-ca-cert\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149539 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.149433 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-certs\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149539 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.149461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-credential-socket\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149539 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.149498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-data\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.149942 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.149916 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istiod-ca-cert\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.150054 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.149165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q7ld\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-kube-api-access-8q7ld\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.150121 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.150093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-socket\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.150182 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.150126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-token\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.150357 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.150338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-socket\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.151483 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.151463 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-envoy\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.151857 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.151834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-podinfo\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.159147 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.159119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q7ld\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-kube-api-access-8q7ld\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.159864 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.159847 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-token\") pod \"e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.291536 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.291426 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:14.579086 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.579053 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp"] Sep 18 21:31:14.580361 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:31:14.580325 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9deb7beb_dcfa_4cbf_904b_6816e91c3183.slice/crio-7103687d9cf75cdc8492bb0607f00a30c8f3e96797dbcea8229e7f85bd1e66c0 WatchSource:0}: Error finding container 7103687d9cf75cdc8492bb0607f00a30c8f3e96797dbcea8229e7f85bd1e66c0: Status 404 returned error can't find the container with id 7103687d9cf75cdc8492bb0607f00a30c8f3e96797dbcea8229e7f85bd1e66c0 Sep 18 21:31:14.582654 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.582623 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:14.582724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.582688 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:14.582724 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.582714 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:14.664611 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:14.664580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" event={"ID":"9deb7beb-dcfa-4cbf-904b-6816e91c3183","Type":"ContainerStarted","Data":"7103687d9cf75cdc8492bb0607f00a30c8f3e96797dbcea8229e7f85bd1e66c0"} Sep 18 21:31:15.672927 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:15.672891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" event={"ID":"9deb7beb-dcfa-4cbf-904b-6816e91c3183","Type":"ContainerStarted","Data":"2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871"} Sep 18 21:31:15.713128 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:15.713073 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" podStartSLOduration=2.7130551990000003 podStartE2EDuration="2.713055199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:31:15.709580951 +0000 UTC m=+1504.639326802" watchObservedRunningTime="2026-09-18 21:31:15.713055199 +0000 UTC m=+1504.642801071" Sep 18 21:31:16.291780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:16.291741 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:16.296904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:16.296881 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:16.679798 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:16.679768 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:16.680701 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:16.680685 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:23.136655 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.136620 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2"] Sep 18 21:31:23.140383 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.140367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:23.142896 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.142874 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8424a6\"" Sep 18 21:31:23.143157 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.143141 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8424a6-dockercfg-8tfrp\"" Sep 18 21:31:23.154888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.154866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2"] Sep 18 21:31:23.245372 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.245331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v48tt\" (UniqueName: \"kubernetes.io/projected/4b842697-d6b1-4a60-add0-f7c393b9e89c-kube-api-access-v48tt\") pod \"payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2\" (UID: \"4b842697-d6b1-4a60-add0-f7c393b9e89c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:23.245722 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.245696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b842697-d6b1-4a60-add0-f7c393b9e89c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2\" (UID: \"4b842697-d6b1-4a60-add0-f7c393b9e89c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:23.346942 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.346898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v48tt\" (UniqueName: \"kubernetes.io/projected/4b842697-d6b1-4a60-add0-f7c393b9e89c-kube-api-access-v48tt\") pod \"payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2\" (UID: \"4b842697-d6b1-4a60-add0-f7c393b9e89c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:23.347155 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.346995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b842697-d6b1-4a60-add0-f7c393b9e89c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2\" (UID: \"4b842697-d6b1-4a60-add0-f7c393b9e89c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:23.347706 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.347680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b842697-d6b1-4a60-add0-f7c393b9e89c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2\" (UID: \"4b842697-d6b1-4a60-add0-f7c393b9e89c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:23.356259 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.356230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v48tt\" (UniqueName: \"kubernetes.io/projected/4b842697-d6b1-4a60-add0-f7c393b9e89c-kube-api-access-v48tt\") pod \"payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2\" (UID: \"4b842697-d6b1-4a60-add0-f7c393b9e89c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:23.452141 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.452063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:23.729837 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:23.729808 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2"] Sep 18 21:31:23.731494 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:31:23.731464 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b842697_d6b1_4a60_add0_f7c393b9e89c.slice/crio-8709e92dbc12598225a52174c4dc43a419481449beb15587ec124b92f43b3c2f WatchSource:0}: Error finding container 8709e92dbc12598225a52174c4dc43a419481449beb15587ec124b92f43b3c2f: Status 404 returned error can't find the container with id 8709e92dbc12598225a52174c4dc43a419481449beb15587ec124b92f43b3c2f Sep 18 21:31:24.736460 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:24.736362 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" event={"ID":"4b842697-d6b1-4a60-add0-f7c393b9e89c","Type":"ContainerStarted","Data":"71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271"} Sep 18 21:31:24.736460 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:24.736430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" event={"ID":"4b842697-d6b1-4a60-add0-f7c393b9e89c","Type":"ContainerStarted","Data":"8709e92dbc12598225a52174c4dc43a419481449beb15587ec124b92f43b3c2f"} Sep 18 21:31:24.736460 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:24.736457 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:24.755907 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:24.755854 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" podStartSLOduration=1.755837734 podStartE2EDuration="1.755837734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:31:24.754338312 +0000 UTC m=+1513.684084186" watchObservedRunningTime="2026-09-18 21:31:24.755837734 +0000 UTC m=+1513.685583607" Sep 18 21:31:35.753192 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:35.753156 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:31:38.838817 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:38.838781 2577 generic.go:356] "Generic (PLEG): container finished" podID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerID="2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871" exitCode=0 Sep 18 21:31:38.839279 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:38.838850 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" event={"ID":"9deb7beb-dcfa-4cbf-904b-6816e91c3183","Type":"ContainerDied","Data":"2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871"} Sep 18 21:31:38.839325 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:38.839313 2577 scope.go:117] "RemoveContainer" containerID="2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871" Sep 18 21:31:38.840474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:38.840450 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:38.840541 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:38.840513 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:38.840582 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:38.840543 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:39.847895 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:39.847859 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" event={"ID":"9deb7beb-dcfa-4cbf-904b-6816e91c3183","Type":"ContainerStarted","Data":"7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b"} Sep 18 21:31:40.292258 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:40.292157 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:40.292507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:40.292482 2577 patch_prober.go:28] interesting pod/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 18 21:31:40.292596 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:40.292543 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 18 21:31:41.292794 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:41.292757 2577 patch_prober.go:28] interesting pod/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 18 21:31:41.293265 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:41.292843 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 18 21:31:42.292715 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:42.292669 2577 patch_prober.go:28] interesting pod/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 18 21:31:42.292928 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:42.292753 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 18 21:31:43.292325 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:43.292290 2577 patch_prober.go:28] interesting pod/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 18 21:31:43.292509 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:43.292348 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 18 21:31:44.291845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:44.291802 2577 patch_prober.go:28] interesting pod/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 18 21:31:44.292426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:44.291865 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 18 21:31:45.291942 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:45.291896 2577 patch_prober.go:28] interesting pod/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 18 21:31:45.292395 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:45.292008 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 18 21:31:46.297340 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:46.297310 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:46.297821 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:46.297650 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:46.298484 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:46.298466 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:31:48.577263 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.577220 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk"] Sep 18 21:31:48.580220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.580196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.583206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.583185 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9e75fb-openshift-default-dockercfg-jkfsd\"" Sep 18 21:31:48.594190 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.594160 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk"] Sep 18 21:31:48.629194 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee10e005-3476-4a74-af72-004006b8c085-istiod-ca-cert\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.629394 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-socket\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.629394 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-istio-token\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.629522 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee10e005-3476-4a74-af72-004006b8c085-istio-podinfo\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.629522 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-data\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.629522 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8l2\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-kube-api-access-dc8l2\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.629698 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-envoy\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.629698 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-certs\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.629698 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.629633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-credential-socket\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731169 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-certs\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-credential-socket\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee10e005-3476-4a74-af72-004006b8c085-istiod-ca-cert\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-socket\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-istio-token\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731615 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-certs\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-socket\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731853 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee10e005-3476-4a74-af72-004006b8c085-istio-podinfo\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731933 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-data\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.731933 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8l2\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-kube-api-access-dc8l2\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.732085 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-credential-socket\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.732085 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.731985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-envoy\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.732197 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.732106 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee10e005-3476-4a74-af72-004006b8c085-istiod-ca-cert\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.732252 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.732216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-data\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.734400 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.734377 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-envoy\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.734771 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.734744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee10e005-3476-4a74-af72-004006b8c085-istio-podinfo\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.749659 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.749624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-istio-token\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.749919 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.749895 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8l2\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-kube-api-access-dc8l2\") pod \"e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:48.891839 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:48.891746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:49.191256 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:49.191222 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk"] Sep 18 21:31:49.194971 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:31:49.194920 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee10e005_3476_4a74_af72_004006b8c085.slice/crio-55ba75eb7da976c6d8cf112ed23eb585e715a53ab762413ab527380dd350b23b WatchSource:0}: Error finding container 55ba75eb7da976c6d8cf112ed23eb585e715a53ab762413ab527380dd350b23b: Status 404 returned error can't find the container with id 55ba75eb7da976c6d8cf112ed23eb585e715a53ab762413ab527380dd350b23b Sep 18 21:31:49.197342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:49.197311 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:49.197424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:49.197375 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:49.197424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:49.197402 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:31:49.922413 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:49.922364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" event={"ID":"ee10e005-3476-4a74-af72-004006b8c085","Type":"ContainerStarted","Data":"fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7"} Sep 18 21:31:49.922927 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:49.922444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" event={"ID":"ee10e005-3476-4a74-af72-004006b8c085","Type":"ContainerStarted","Data":"55ba75eb7da976c6d8cf112ed23eb585e715a53ab762413ab527380dd350b23b"} Sep 18 21:31:50.685780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.685672 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" podStartSLOduration=2.685654452 podStartE2EDuration="2.685654452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:31:49.987411368 +0000 UTC m=+1538.917157243" watchObservedRunningTime="2026-09-18 21:31:50.685654452 +0000 UTC m=+1539.615400324" Sep 18 21:31:50.687127 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.687107 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7870-117f9a"] Sep 18 21:31:50.690026 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.690010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7870-117f9a" Sep 18 21:31:50.692392 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.692365 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrg6b\"" Sep 18 21:31:50.697025 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.696996 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7870-117f9a"] Sep 18 21:31:50.755495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.755458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqnb\" (UniqueName: \"kubernetes.io/projected/77f814ae-1200-4ebc-893e-b323e9969ffc-kube-api-access-bwqnb\") pod \"test-curl-7870-117f9a\" (UID: \"77f814ae-1200-4ebc-893e-b323e9969ffc\") " pod="opendatahub/test-curl-7870-117f9a" Sep 18 21:31:50.856823 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.856784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqnb\" (UniqueName: \"kubernetes.io/projected/77f814ae-1200-4ebc-893e-b323e9969ffc-kube-api-access-bwqnb\") pod \"test-curl-7870-117f9a\" (UID: \"77f814ae-1200-4ebc-893e-b323e9969ffc\") " pod="opendatahub/test-curl-7870-117f9a" Sep 18 21:31:50.879464 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.879436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqnb\" (UniqueName: \"kubernetes.io/projected/77f814ae-1200-4ebc-893e-b323e9969ffc-kube-api-access-bwqnb\") pod \"test-curl-7870-117f9a\" (UID: \"77f814ae-1200-4ebc-893e-b323e9969ffc\") " pod="opendatahub/test-curl-7870-117f9a" Sep 18 21:31:50.892332 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.892301 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:50.898366 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.898332 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:50.929723 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.929692 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:50.931116 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:50.931091 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:31:51.002663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:51.002561 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7870-117f9a" Sep 18 21:31:51.291052 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:51.291028 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7870-117f9a"] Sep 18 21:31:51.292009 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:31:51.291976 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77f814ae_1200_4ebc_893e_b323e9969ffc.slice/crio-da0a19c8745f1e9a48a363a7fc4ea1d1429974c5be31a666c35e603e2aec247e WatchSource:0}: Error finding container da0a19c8745f1e9a48a363a7fc4ea1d1429974c5be31a666c35e603e2aec247e: Status 404 returned error can't find the container with id da0a19c8745f1e9a48a363a7fc4ea1d1429974c5be31a666c35e603e2aec247e Sep 18 21:31:51.939721 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:51.939686 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7870-117f9a" event={"ID":"77f814ae-1200-4ebc-893e-b323e9969ffc","Type":"ContainerStarted","Data":"1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c"} Sep 18 21:31:51.940196 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:51.939734 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7870-117f9a" event={"ID":"77f814ae-1200-4ebc-893e-b323e9969ffc","Type":"ContainerStarted","Data":"da0a19c8745f1e9a48a363a7fc4ea1d1429974c5be31a666c35e603e2aec247e"} Sep 18 21:31:51.970137 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:51.970079 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7870-117f9a" podStartSLOduration=1.9700604720000001 podStartE2EDuration="1.970060472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:31:51.968931967 +0000 UTC m=+1540.898677841" watchObservedRunningTime="2026-09-18 21:31:51.970060472 +0000 UTC m=+1540.899806408" Sep 18 21:31:52.635134 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:52.635046 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7870-117f9a"] Sep 18 21:31:52.651079 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:52.651039 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7870-117f9a"] Sep 18 21:31:52.947545 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:31:52.947451 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7870-117f9a" container="test-curl-7870-117f9a" Sep 18 21:31:53.954653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:53.954596 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7870-117f9a" podUID="77f814ae-1200-4ebc-893e-b323e9969ffc" containerName="test-curl-7870-117f9a" containerID="cri-o://1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c" gracePeriod=2 Sep 18 21:31:56.148571 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.148539 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7870-117f9a" Sep 18 21:31:56.219644 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.219549 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwqnb\" (UniqueName: \"kubernetes.io/projected/77f814ae-1200-4ebc-893e-b323e9969ffc-kube-api-access-bwqnb\") pod \"77f814ae-1200-4ebc-893e-b323e9969ffc\" (UID: \"77f814ae-1200-4ebc-893e-b323e9969ffc\") " Sep 18 21:31:56.221625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.221596 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f814ae-1200-4ebc-893e-b323e9969ffc-kube-api-access-bwqnb" (OuterVolumeSpecName: "kube-api-access-bwqnb") pod "77f814ae-1200-4ebc-893e-b323e9969ffc" (UID: "77f814ae-1200-4ebc-893e-b323e9969ffc"). InnerVolumeSpecName "kube-api-access-bwqnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:31:56.321064 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.321023 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwqnb\" (UniqueName: \"kubernetes.io/projected/77f814ae-1200-4ebc-893e-b323e9969ffc-kube-api-access-bwqnb\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:31:56.976305 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.976262 2577 generic.go:356] "Generic (PLEG): container finished" podID="77f814ae-1200-4ebc-893e-b323e9969ffc" containerID="1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c" exitCode=137 Sep 18 21:31:56.976305 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.976307 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7870-117f9a" Sep 18 21:31:56.976585 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.976321 2577 scope.go:117] "RemoveContainer" containerID="1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c" Sep 18 21:31:56.986166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.986142 2577 scope.go:117] "RemoveContainer" containerID="1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c" Sep 18 21:31:56.986465 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:31:56.986447 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c\": container with ID starting with 1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c not found: ID does not exist" containerID="1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c" Sep 18 21:31:56.986518 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:56.986477 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c"} err="failed to get container status \"1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c\": rpc error: code = NotFound desc = could not find container \"1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c\": container with ID starting with 1dade142feaddc767160e81313130ae89fac60c813bd87387a946dc6b5b6e33c not found: ID does not exist" Sep 18 21:31:56.986913 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:31:56.986884 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7870-117f9a\" is forbidden: User \"system:node:ip-10-0-132-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-187.ec2.internal' and this object" podUID="77f814ae-1200-4ebc-893e-b323e9969ffc" pod="opendatahub/test-curl-7870-117f9a" Sep 18 21:31:57.360712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:31:57.360675 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f814ae-1200-4ebc-893e-b323e9969ffc" path="/var/lib/kubelet/pods/77f814ae-1200-4ebc-893e-b323e9969ffc/volumes" Sep 18 21:32:16.136276 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.136230 2577 generic.go:356] "Generic (PLEG): container finished" podID="ee10e005-3476-4a74-af72-004006b8c085" containerID="fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7" exitCode=0 Sep 18 21:32:16.136819 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.136407 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" event={"ID":"ee10e005-3476-4a74-af72-004006b8c085","Type":"ContainerDied","Data":"fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7"} Sep 18 21:32:16.137125 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.137103 2577 scope.go:117] "RemoveContainer" containerID="fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7" Sep 18 21:32:16.139412 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.139369 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:32:16.139567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.139537 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:32:16.139799 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.139781 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:32:16.676230 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.676189 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7"] Sep 18 21:32:16.676702 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.676689 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f814ae-1200-4ebc-893e-b323e9969ffc" containerName="test-curl-7870-117f9a" Sep 18 21:32:16.676757 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.676704 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f814ae-1200-4ebc-893e-b323e9969ffc" containerName="test-curl-7870-117f9a" Sep 18 21:32:16.676793 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.676781 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f814ae-1200-4ebc-893e-b323e9969ffc" containerName="test-curl-7870-117f9a" Sep 18 21:32:16.679154 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.679133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.684845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.684800 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cad5cf\"/\"sub-test-model-cad5cf-kserve-self-signed-certs\"" Sep 18 21:32:16.687749 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.687728 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cad5cf\"/\"default-dockercfg-8qzpg\"" Sep 18 21:32:16.690616 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.690597 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cad5cf\"/\"kube-root-ca.crt\"" Sep 18 21:32:16.691390 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.691370 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cad5cf\"/\"openshift-service-ca.crt\"" Sep 18 21:32:16.732975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.732918 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7"] Sep 18 21:32:16.748865 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.748832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4658f41a-d017-41c3-a15d-978403bca4d9-tls-certs\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.749083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.748877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-home\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.749083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.748996 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-model-cache\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.749083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.749025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-dshm\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.749083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.749050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxsr\" (UniqueName: \"kubernetes.io/projected/4658f41a-d017-41c3-a15d-978403bca4d9-kube-api-access-lrxsr\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.749306 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.749152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-tmp-dir\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.850639 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.850587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4658f41a-d017-41c3-a15d-978403bca4d9-tls-certs\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.850862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.850660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-home\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.850862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.850725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-model-cache\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.850862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.850758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-dshm\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.850862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.850784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxsr\" (UniqueName: \"kubernetes.io/projected/4658f41a-d017-41c3-a15d-978403bca4d9-kube-api-access-lrxsr\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.850862 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.850818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-tmp-dir\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.851451 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.851419 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-tmp-dir\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.851803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.851781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-model-cache\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.853228 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.853206 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-home\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.855285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.855257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-dshm\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.855806 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.855773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4658f41a-d017-41c3-a15d-978403bca4d9-tls-certs\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.884250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.884215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxsr\" (UniqueName: \"kubernetes.io/projected/4658f41a-d017-41c3-a15d-978403bca4d9-kube-api-access-lrxsr\") pod \"sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:16.988834 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:16.988733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:17.157841 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:17.157800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" event={"ID":"ee10e005-3476-4a74-af72-004006b8c085","Type":"ContainerStarted","Data":"29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d"} Sep 18 21:32:17.301576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:17.301537 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7"] Sep 18 21:32:17.303405 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:32:17.303370 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4658f41a_d017_41c3_a15d_978403bca4d9.slice/crio-a90fc4675f71c2f62f3984c626d3a277b3921d16ac8ed8b68da83bfc8341aee2 WatchSource:0}: Error finding container a90fc4675f71c2f62f3984c626d3a277b3921d16ac8ed8b68da83bfc8341aee2: Status 404 returned error can't find the container with id a90fc4675f71c2f62f3984c626d3a277b3921d16ac8ed8b68da83bfc8341aee2 Sep 18 21:32:17.891948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:17.891904 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:32:17.893358 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:17.893328 2577 patch_prober.go:28] interesting pod/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 21:32:17.893496 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:17.893385 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 21:32:18.166134 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:18.166031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" event={"ID":"4658f41a-d017-41c3-a15d-978403bca4d9","Type":"ContainerStarted","Data":"cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc"} Sep 18 21:32:18.166134 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:18.166082 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" event={"ID":"4658f41a-d017-41c3-a15d-978403bca4d9","Type":"ContainerStarted","Data":"a90fc4675f71c2f62f3984c626d3a277b3921d16ac8ed8b68da83bfc8341aee2"} Sep 18 21:32:18.186903 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:18.186842 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" podStartSLOduration=2.186821618 podStartE2EDuration="2.186821618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:32:18.183360644 +0000 UTC m=+1567.113106518" watchObservedRunningTime="2026-09-18 21:32:18.186821618 +0000 UTC m=+1567.116567491" Sep 18 21:32:18.892145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:18.892101 2577 patch_prober.go:28] interesting pod/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 21:32:18.892322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:18.892176 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 21:32:19.893063 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:19.893023 2577 patch_prober.go:28] interesting pod/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 21:32:19.893465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:19.893089 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 21:32:20.893124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:20.893084 2577 patch_prober.go:28] interesting pod/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 21:32:20.893505 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:20.893158 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 21:32:20.931488 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:20.930464 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:32:21.892982 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:21.892919 2577 patch_prober.go:28] interesting pod/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 21:32:21.893183 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:21.893013 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 21:32:22.896512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:22.896481 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:32:22.897680 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:22.897659 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:32:25.542068 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.542033 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s"] Sep 18 21:32:25.544894 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.544875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" Sep 18 21:32:25.547270 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.547242 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 21:32:25.547417 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.547281 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-52bhz\"" Sep 18 21:32:25.558106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.555509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s"] Sep 18 21:32:25.657426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.657395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67196fb2-b662-4883-982d-e89888f1b609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s\" (UID: \"67196fb2-b662-4883-982d-e89888f1b609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" Sep 18 21:32:25.758111 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.758069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67196fb2-b662-4883-982d-e89888f1b609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s\" (UID: \"67196fb2-b662-4883-982d-e89888f1b609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" Sep 18 21:32:25.758726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.758702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67196fb2-b662-4883-982d-e89888f1b609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s\" (UID: \"67196fb2-b662-4883-982d-e89888f1b609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" Sep 18 21:32:25.862617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:25.862581 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" Sep 18 21:32:26.137199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:26.137160 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s"] Sep 18 21:32:26.138310 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:32:26.138272 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67196fb2_b662_4883_982d_e89888f1b609.slice/crio-5d47fb804ddc333f02272e5eedcb5843f7b6ac96cc380488ccd8099c75014f3e WatchSource:0}: Error finding container 5d47fb804ddc333f02272e5eedcb5843f7b6ac96cc380488ccd8099c75014f3e: Status 404 returned error can't find the container with id 5d47fb804ddc333f02272e5eedcb5843f7b6ac96cc380488ccd8099c75014f3e Sep 18 21:32:26.221653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:26.221610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" event={"ID":"67196fb2-b662-4883-982d-e89888f1b609","Type":"ContainerStarted","Data":"5d47fb804ddc333f02272e5eedcb5843f7b6ac96cc380488ccd8099c75014f3e"} Sep 18 21:32:26.989117 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:26.989075 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:26.989572 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:26.989179 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:27.002083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:27.002048 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:27.232758 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:27.232650 2577 generic.go:356] "Generic (PLEG): container finished" podID="67196fb2-b662-4883-982d-e89888f1b609" containerID="cd7733e38eb05460931aed4c954189e812ca3c6bc80f0383f960ee618234a6a8" exitCode=0 Sep 18 21:32:27.232930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:27.232746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" event={"ID":"67196fb2-b662-4883-982d-e89888f1b609","Type":"ContainerDied","Data":"cd7733e38eb05460931aed4c954189e812ca3c6bc80f0383f960ee618234a6a8"} Sep 18 21:32:27.244856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:27.244820 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:32:28.411130 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:28.411104 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" Sep 18 21:32:28.486857 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:28.486819 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67196fb2-b662-4883-982d-e89888f1b609-maas-api-service-ca\") pod \"67196fb2-b662-4883-982d-e89888f1b609\" (UID: \"67196fb2-b662-4883-982d-e89888f1b609\") " Sep 18 21:32:28.487239 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:28.487215 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67196fb2-b662-4883-982d-e89888f1b609-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67196fb2-b662-4883-982d-e89888f1b609" (UID: "67196fb2-b662-4883-982d-e89888f1b609"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:32:28.588482 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:28.588443 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67196fb2-b662-4883-982d-e89888f1b609-maas-api-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:29.247167 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:29.247134 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" Sep 18 21:32:29.247376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:29.247168 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s" event={"ID":"67196fb2-b662-4883-982d-e89888f1b609","Type":"ContainerDied","Data":"5d47fb804ddc333f02272e5eedcb5843f7b6ac96cc380488ccd8099c75014f3e"} Sep 18 21:32:29.247376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:29.247211 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d47fb804ddc333f02272e5eedcb5843f7b6ac96cc380488ccd8099c75014f3e" Sep 18 21:32:41.481224 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.481189 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2"] Sep 18 21:32:41.481688 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.481430 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" podUID="4b842697-d6b1-4a60-add0-f7c393b9e89c" containerName="payload-pre-processing" containerID="cri-o://71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271" gracePeriod=30 Sep 18 21:32:41.661376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.661341 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s"] Sep 18 21:32:41.666434 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.666403 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8424a6-xlx4s"] Sep 18 21:32:41.752253 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.752226 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:32:41.841093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.840936 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v48tt\" (UniqueName: \"kubernetes.io/projected/4b842697-d6b1-4a60-add0-f7c393b9e89c-kube-api-access-v48tt\") pod \"4b842697-d6b1-4a60-add0-f7c393b9e89c\" (UID: \"4b842697-d6b1-4a60-add0-f7c393b9e89c\") " Sep 18 21:32:41.841093 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.841083 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b842697-d6b1-4a60-add0-f7c393b9e89c-plugins-config-volume\") pod \"4b842697-d6b1-4a60-add0-f7c393b9e89c\" (UID: \"4b842697-d6b1-4a60-add0-f7c393b9e89c\") " Sep 18 21:32:41.841701 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.841635 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b842697-d6b1-4a60-add0-f7c393b9e89c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b842697-d6b1-4a60-add0-f7c393b9e89c" (UID: "4b842697-d6b1-4a60-add0-f7c393b9e89c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:32:41.842344 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.842312 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b842697-d6b1-4a60-add0-f7c393b9e89c-plugins-config-volume\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:41.844727 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.844696 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b842697-d6b1-4a60-add0-f7c393b9e89c-kube-api-access-v48tt" (OuterVolumeSpecName: "kube-api-access-v48tt") pod "4b842697-d6b1-4a60-add0-f7c393b9e89c" (UID: "4b842697-d6b1-4a60-add0-f7c393b9e89c"). InnerVolumeSpecName "kube-api-access-v48tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:32:41.942936 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:41.942901 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v48tt\" (UniqueName: \"kubernetes.io/projected/4b842697-d6b1-4a60-add0-f7c393b9e89c-kube-api-access-v48tt\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:42.332333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.332296 2577 generic.go:356] "Generic (PLEG): container finished" podID="4b842697-d6b1-4a60-add0-f7c393b9e89c" containerID="71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271" exitCode=0 Sep 18 21:32:42.332514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.332360 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" Sep 18 21:32:42.332514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.332386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" event={"ID":"4b842697-d6b1-4a60-add0-f7c393b9e89c","Type":"ContainerDied","Data":"71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271"} Sep 18 21:32:42.332514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.332446 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2" event={"ID":"4b842697-d6b1-4a60-add0-f7c393b9e89c","Type":"ContainerDied","Data":"8709e92dbc12598225a52174c4dc43a419481449beb15587ec124b92f43b3c2f"} Sep 18 21:32:42.332514 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.332461 2577 scope.go:117] "RemoveContainer" containerID="71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271" Sep 18 21:32:42.342975 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.342933 2577 scope.go:117] "RemoveContainer" containerID="71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271" Sep 18 21:32:42.343275 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:32:42.343255 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271\": container with ID starting with 71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271 not found: ID does not exist" containerID="71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271" Sep 18 21:32:42.343357 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.343284 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271"} err="failed to get container status \"71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271\": rpc error: code = NotFound desc = could not find container \"71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271\": container with ID starting with 71f6f5f435f99d274a64e19bef11ca16d5ccb8db92e6909f6dc0fa2894b45271 not found: ID does not exist" Sep 18 21:32:42.358973 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.358921 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2"] Sep 18 21:32:42.361876 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:42.361845 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8424a6-746c749f9d-szhc2"] Sep 18 21:32:43.359660 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:43.359616 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b842697-d6b1-4a60-add0-f7c393b9e89c" path="/var/lib/kubelet/pods/4b842697-d6b1-4a60-add0-f7c393b9e89c/volumes" Sep 18 21:32:43.360199 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:43.360184 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67196fb2-b662-4883-982d-e89888f1b609" path="/var/lib/kubelet/pods/67196fb2-b662-4883-982d-e89888f1b609/volumes" Sep 18 21:32:49.333762 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.333725 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp"] Sep 18 21:32:49.334287 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.334067 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" containerID="cri-o://7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b" gracePeriod=30 Sep 18 21:32:49.696404 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.696297 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf"] Sep 18 21:32:49.696858 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.696837 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67196fb2-b662-4883-982d-e89888f1b609" containerName="revoke-keys" Sep 18 21:32:49.697002 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.696861 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67196fb2-b662-4883-982d-e89888f1b609" containerName="revoke-keys" Sep 18 21:32:49.697002 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.696897 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b842697-d6b1-4a60-add0-f7c393b9e89c" containerName="payload-pre-processing" Sep 18 21:32:49.697002 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.696906 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b842697-d6b1-4a60-add0-f7c393b9e89c" containerName="payload-pre-processing" Sep 18 21:32:49.697181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.697057 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b842697-d6b1-4a60-add0-f7c393b9e89c" containerName="payload-pre-processing" Sep 18 21:32:49.697181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.697073 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="67196fb2-b662-4883-982d-e89888f1b609" containerName="revoke-keys" Sep 18 21:32:49.701301 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.701279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" Sep 18 21:32:49.703772 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.703746 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 21:32:49.703894 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.703746 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-52bhz\"" Sep 18 21:32:49.711004 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.710946 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf"] Sep 18 21:32:49.841871 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.841830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4786713e-acfd-4fa6-a80b-e0d053295437-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf\" (UID: \"4786713e-acfd-4fa6-a80b-e0d053295437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" Sep 18 21:32:49.942776 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.942732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4786713e-acfd-4fa6-a80b-e0d053295437-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf\" (UID: \"4786713e-acfd-4fa6-a80b-e0d053295437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" Sep 18 21:32:49.943467 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:49.943441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4786713e-acfd-4fa6-a80b-e0d053295437-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf\" (UID: \"4786713e-acfd-4fa6-a80b-e0d053295437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" Sep 18 21:32:50.016485 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:50.016383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" Sep 18 21:32:50.306795 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:50.306764 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf"] Sep 18 21:32:50.307914 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:32:50.307883 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4786713e_acfd_4fa6_a80b_e0d053295437.slice/crio-5cfacd364cfb19326e44415c300b374822fd97596b8cf8b9696945fcff70fbe9 WatchSource:0}: Error finding container 5cfacd364cfb19326e44415c300b374822fd97596b8cf8b9696945fcff70fbe9: Status 404 returned error can't find the container with id 5cfacd364cfb19326e44415c300b374822fd97596b8cf8b9696945fcff70fbe9 Sep 18 21:32:50.394801 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:50.394764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" event={"ID":"4786713e-acfd-4fa6-a80b-e0d053295437","Type":"ContainerStarted","Data":"5cfacd364cfb19326e44415c300b374822fd97596b8cf8b9696945fcff70fbe9"} Sep 18 21:32:51.403011 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:51.402974 2577 generic.go:356] "Generic (PLEG): container finished" podID="4786713e-acfd-4fa6-a80b-e0d053295437" containerID="c2ed57e809282fe2715f61d40817b7113d198602e681cc7d040ff5aa7e305ca9" exitCode=0 Sep 18 21:32:51.403414 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:51.403058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" event={"ID":"4786713e-acfd-4fa6-a80b-e0d053295437","Type":"ContainerDied","Data":"c2ed57e809282fe2715f61d40817b7113d198602e681cc7d040ff5aa7e305ca9"} Sep 18 21:32:52.554854 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:52.554820 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" Sep 18 21:32:52.676233 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:52.676145 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4786713e-acfd-4fa6-a80b-e0d053295437-maas-api-service-ca\") pod \"4786713e-acfd-4fa6-a80b-e0d053295437\" (UID: \"4786713e-acfd-4fa6-a80b-e0d053295437\") " Sep 18 21:32:52.676584 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:52.676556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4786713e-acfd-4fa6-a80b-e0d053295437-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4786713e-acfd-4fa6-a80b-e0d053295437" (UID: "4786713e-acfd-4fa6-a80b-e0d053295437"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:32:52.777055 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:52.777004 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4786713e-acfd-4fa6-a80b-e0d053295437-maas-api-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:53.417559 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:53.417528 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" Sep 18 21:32:53.417718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:53.417525 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf" event={"ID":"4786713e-acfd-4fa6-a80b-e0d053295437","Type":"ContainerDied","Data":"5cfacd364cfb19326e44415c300b374822fd97596b8cf8b9696945fcff70fbe9"} Sep 18 21:32:53.417718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:53.417648 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cfacd364cfb19326e44415c300b374822fd97596b8cf8b9696945fcff70fbe9" Sep 18 21:32:54.600202 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.600117 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:32:54.697901 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.697866 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-data\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.697901 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.697905 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-token\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.698151 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.697929 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-credential-socket\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.698151 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.697948 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-socket\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.698151 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698088 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-podinfo\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.698320 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q7ld\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-kube-api-access-8q7ld\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.698320 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698194 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istiod-ca-cert\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.698320 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698227 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:32:54.698320 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698241 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-data" (OuterVolumeSpecName: "istio-data") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:32:54.698320 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698279 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:32:54.698320 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698299 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-envoy\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.698624 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698348 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-certs\") pod \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\" (UID: \"9deb7beb-dcfa-4cbf-904b-6816e91c3183\") " Sep 18 21:32:54.698624 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698521 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:32:54.698738 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698670 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:32:54.698835 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698812 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:54.698835 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698835 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:54.699084 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698848 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:54.699084 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698861 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:54.699084 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.698875 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:54.700532 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.700490 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:32:54.700637 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.700582 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-token" (OuterVolumeSpecName: "istio-token") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:32:54.700708 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.700686 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-kube-api-access-8q7ld" (OuterVolumeSpecName: "kube-api-access-8q7ld") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "kube-api-access-8q7ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:32:54.700872 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.700848 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9deb7beb-dcfa-4cbf-904b-6816e91c3183" (UID: "9deb7beb-dcfa-4cbf-904b-6816e91c3183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:32:54.800253 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.800218 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:54.800253 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.800249 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:54.800253 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.800259 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deb7beb-dcfa-4cbf-904b-6816e91c3183-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:54.800503 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:54.800270 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q7ld\" (UniqueName: \"kubernetes.io/projected/9deb7beb-dcfa-4cbf-904b-6816e91c3183-kube-api-access-8q7ld\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:32:55.434146 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.434107 2577 generic.go:356] "Generic (PLEG): container finished" podID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerID="7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b" exitCode=0 Sep 18 21:32:55.434322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.434164 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" Sep 18 21:32:55.434322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.434188 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" event={"ID":"9deb7beb-dcfa-4cbf-904b-6816e91c3183","Type":"ContainerDied","Data":"7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b"} Sep 18 21:32:55.434322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.434247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp" event={"ID":"9deb7beb-dcfa-4cbf-904b-6816e91c3183","Type":"ContainerDied","Data":"7103687d9cf75cdc8492bb0607f00a30c8f3e96797dbcea8229e7f85bd1e66c0"} Sep 18 21:32:55.434322 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.434271 2577 scope.go:117] "RemoveContainer" containerID="7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b" Sep 18 21:32:55.444826 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.444795 2577 scope.go:117] "RemoveContainer" containerID="2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871" Sep 18 21:32:55.456384 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.455382 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp"] Sep 18 21:32:55.458005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.457795 2577 scope.go:117] "RemoveContainer" containerID="7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b" Sep 18 21:32:55.458224 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:32:55.458196 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b\": container with ID starting with 7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b not found: ID does not exist" containerID="7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b" Sep 18 21:32:55.458313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.458247 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b"} err="failed to get container status \"7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b\": rpc error: code = NotFound desc = could not find container \"7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b\": container with ID starting with 7ab82e14e527c8158d65bff86caea49aa527b03670f05d346d1c38529131199b not found: ID does not exist" Sep 18 21:32:55.458313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.458277 2577 scope.go:117] "RemoveContainer" containerID="2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871" Sep 18 21:32:55.458673 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:32:55.458640 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871\": container with ID starting with 2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871 not found: ID does not exist" containerID="2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871" Sep 18 21:32:55.458786 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.458679 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871"} err="failed to get container status \"2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871\": rpc error: code = NotFound desc = could not find container \"2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871\": container with ID starting with 2de1e4a819dc096a0435e31d172f3fc0f2c1e442148d85479e04fb68f2a7e871 not found: ID does not exist" Sep 18 21:32:55.462391 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:55.462369 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8424a6-openshift-default-77cd9699cd-cg9qp"] Sep 18 21:32:57.360061 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:32:57.360028 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" path="/var/lib/kubelet/pods/9deb7beb-dcfa-4cbf-904b-6816e91c3183/volumes" Sep 18 21:33:05.453345 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:05.453300 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf"] Sep 18 21:33:05.457551 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:05.457521 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9e75fb-bcgdf"] Sep 18 21:33:07.361009 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:07.360944 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4786713e-acfd-4fa6-a80b-e0d053295437" path="/var/lib/kubelet/pods/4786713e-acfd-4fa6-a80b-e0d053295437/volumes" Sep 18 21:33:12.553850 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:12.553809 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk"] Sep 18 21:33:12.554342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:12.554083 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" containerID="cri-o://29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d" gracePeriod=30 Sep 18 21:33:17.821275 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.821207 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:33:17.953563 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953524 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee10e005-3476-4a74-af72-004006b8c085-istio-podinfo\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.953761 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953588 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee10e005-3476-4a74-af72-004006b8c085-istiod-ca-cert\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.953761 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953645 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-data\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.953761 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953660 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-credential-socket\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.953891 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953782 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-istio-token\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.953937 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953901 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-envoy\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.953937 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953907 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:17.954106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953933 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-certs\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.954106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.953983 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-socket\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.954106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954003 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:17.954106 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954012 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc8l2\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-kube-api-access-dc8l2\") pod \"ee10e005-3476-4a74-af72-004006b8c085\" (UID: \"ee10e005-3476-4a74-af72-004006b8c085\") " Sep 18 21:33:17.954333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954232 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:17.954333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954255 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee10e005-3476-4a74-af72-004006b8c085-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:33:17.954512 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954486 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:17.954609 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954514 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee10e005-3476-4a74-af72-004006b8c085-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:17.954609 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954527 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:17.954609 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954523 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:17.954609 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.954538 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:17.956102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.956069 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee10e005-3476-4a74-af72-004006b8c085-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:33:17.956223 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.956187 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-kube-api-access-dc8l2" (OuterVolumeSpecName: "kube-api-access-dc8l2") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "kube-api-access-dc8l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:33:17.956409 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.956381 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:33:17.956409 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:17.956395 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee10e005-3476-4a74-af72-004006b8c085" (UID: "ee10e005-3476-4a74-af72-004006b8c085"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:18.056102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.056058 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:18.056102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.056093 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:18.056102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.056102 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee10e005-3476-4a74-af72-004006b8c085-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:18.056102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.056111 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc8l2\" (UniqueName: \"kubernetes.io/projected/ee10e005-3476-4a74-af72-004006b8c085-kube-api-access-dc8l2\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:18.056416 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.056122 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee10e005-3476-4a74-af72-004006b8c085-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:18.603638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.603604 2577 generic.go:356] "Generic (PLEG): container finished" podID="ee10e005-3476-4a74-af72-004006b8c085" containerID="29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d" exitCode=0 Sep 18 21:33:18.603824 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.603653 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" Sep 18 21:33:18.603824 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.603687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" event={"ID":"ee10e005-3476-4a74-af72-004006b8c085","Type":"ContainerDied","Data":"29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d"} Sep 18 21:33:18.603824 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.603741 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk" event={"ID":"ee10e005-3476-4a74-af72-004006b8c085","Type":"ContainerDied","Data":"55ba75eb7da976c6d8cf112ed23eb585e715a53ab762413ab527380dd350b23b"} Sep 18 21:33:18.603824 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.603765 2577 scope.go:117] "RemoveContainer" containerID="29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d" Sep 18 21:33:18.614559 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.614535 2577 scope.go:117] "RemoveContainer" containerID="fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7" Sep 18 21:33:18.624835 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.624783 2577 scope.go:117] "RemoveContainer" containerID="29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d" Sep 18 21:33:18.625233 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:33:18.625208 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d\": container with ID starting with 29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d not found: ID does not exist" containerID="29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d" Sep 18 21:33:18.625348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.625242 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d"} err="failed to get container status \"29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d\": rpc error: code = NotFound desc = could not find container \"29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d\": container with ID starting with 29cca3b3d140a482f9ace5e0ec7ce22d36fc83e5f400d764b4d588e843995a7d not found: ID does not exist" Sep 18 21:33:18.625348 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.625260 2577 scope.go:117] "RemoveContainer" containerID="fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7" Sep 18 21:33:18.625557 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:33:18.625532 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7\": container with ID starting with fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7 not found: ID does not exist" containerID="fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7" Sep 18 21:33:18.625623 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.625565 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7"} err="failed to get container status \"fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7\": rpc error: code = NotFound desc = could not find container \"fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7\": container with ID starting with fd51c23e5c768a99e3638b9c721a3dfe5db516d1aa1458c8ff14bd9a9d340ad7 not found: ID does not exist" Sep 18 21:33:18.629123 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.629098 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk"] Sep 18 21:33:18.631926 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:18.631904 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9e75fb-openshift-default-d9476d49d-4k4mk"] Sep 18 21:33:19.359461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:19.359424 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee10e005-3476-4a74-af72-004006b8c085" path="/var/lib/kubelet/pods/ee10e005-3476-4a74-af72-004006b8c085/volumes" Sep 18 21:33:23.866585 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:23.866545 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k"] Sep 18 21:33:23.872799 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:23.872769 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1b65ac-29829450-cdq4k"] Sep 18 21:33:25.360825 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:25.360780 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="914e186c-5fc3-4084-8de2-6006ef81b90f" path="/var/lib/kubelet/pods/914e186c-5fc3-4084-8de2-6006ef81b90f/volumes" Sep 18 21:33:30.651672 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:30.651631 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh"] Sep 18 21:33:30.652257 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:30.651905 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" containerID="cri-o://0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f" gracePeriod=30 Sep 18 21:33:35.914686 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.914615 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:33:35.944381 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944343 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8652\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-kube-api-access-m8652\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.944381 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944382 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istiod-ca-cert\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.944625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944415 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-socket\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.944625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944455 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-envoy\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.944625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944490 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-certs\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.944625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944511 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-token\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.944625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944541 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-credential-socket\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.944900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944778 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:35.944900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944789 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:35.944900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944814 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-data\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.944900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944827 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:33:35.944900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.944875 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-podinfo\") pod \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\" (UID: \"c20cb642-c603-4d97-8e6f-0ec218d3fa0b\") " Sep 18 21:33:35.945189 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.945022 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:35.945189 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.945124 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:35.945358 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.945343 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:35.945410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.945361 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:35.945410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.945371 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:35.945410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.945381 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:35.945410 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.945389 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:35.947299 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.947266 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-kube-api-access-m8652" (OuterVolumeSpecName: "kube-api-access-m8652") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "kube-api-access-m8652". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:33:35.947299 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.947283 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:35.947299 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.947293 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:33:35.947475 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:35.947301 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c20cb642-c603-4d97-8e6f-0ec218d3fa0b" (UID: "c20cb642-c603-4d97-8e6f-0ec218d3fa0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:33:36.046398 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.046349 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:36.046398 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.046389 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:36.046398 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.046397 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:36.046398 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.046406 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8652\" (UniqueName: \"kubernetes.io/projected/c20cb642-c603-4d97-8e6f-0ec218d3fa0b-kube-api-access-m8652\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:36.728440 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.728401 2577 generic.go:356] "Generic (PLEG): container finished" podID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerID="0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f" exitCode=0 Sep 18 21:33:36.728440 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.728452 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" Sep 18 21:33:36.728722 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.728490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" event={"ID":"c20cb642-c603-4d97-8e6f-0ec218d3fa0b","Type":"ContainerDied","Data":"0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f"} Sep 18 21:33:36.728722 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.728545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh" event={"ID":"c20cb642-c603-4d97-8e6f-0ec218d3fa0b","Type":"ContainerDied","Data":"6eccad966e1608f694a83a6aa76fac48070aaf138181ed4d46ebe4a7e32ae433"} Sep 18 21:33:36.728722 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.728569 2577 scope.go:117] "RemoveContainer" containerID="0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f" Sep 18 21:33:36.739368 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.739343 2577 scope.go:117] "RemoveContainer" containerID="af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa" Sep 18 21:33:36.748966 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.748928 2577 scope.go:117] "RemoveContainer" containerID="0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f" Sep 18 21:33:36.749272 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:33:36.749252 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f\": container with ID starting with 0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f not found: ID does not exist" containerID="0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f" Sep 18 21:33:36.749332 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.749281 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f"} err="failed to get container status \"0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f\": rpc error: code = NotFound desc = could not find container \"0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f\": container with ID starting with 0a00663ae9cb4b8ef5ff23b966b12fe36e1acc0d0e21e98b87dbc70acbea846f not found: ID does not exist" Sep 18 21:33:36.749332 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.749299 2577 scope.go:117] "RemoveContainer" containerID="af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa" Sep 18 21:33:36.749590 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:33:36.749572 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa\": container with ID starting with af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa not found: ID does not exist" containerID="af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa" Sep 18 21:33:36.749663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.749599 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa"} err="failed to get container status \"af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa\": rpc error: code = NotFound desc = could not find container \"af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa\": container with ID starting with af0da362e12a7afbd1f7511158a860c1a82bfe75c5fa67ec70c94374d1ab4cfa not found: ID does not exist" Sep 18 21:33:36.758243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.758213 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh"] Sep 18 21:33:36.764345 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:36.764319 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1b65ac-openshift-default-78cf99894-h4dlh"] Sep 18 21:33:37.360755 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:37.360717 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" path="/var/lib/kubelet/pods/c20cb642-c603-4d97-8e6f-0ec218d3fa0b/volumes" Sep 18 21:33:37.854485 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:37.854454 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7"] Sep 18 21:33:41.739663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:41.739630 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq"] Sep 18 21:33:41.742312 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:41.742283 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-514a1a-29829450-xk8hq"] Sep 18 21:33:43.359661 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:43.359624 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb57796-24f6-48eb-a56e-3f79e1f0e0f6" path="/var/lib/kubelet/pods/4fb57796-24f6-48eb-a56e-3f79e1f0e0f6/volumes" Sep 18 21:33:52.874974 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:52.874917 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" podUID="4658f41a-d017-41c3-a15d-978403bca4d9" containerName="main" containerID="cri-o://cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc" gracePeriod=45 Sep 18 21:33:53.156041 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.156013 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:33:53.211339 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211300 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-tmp-dir\") pod \"4658f41a-d017-41c3-a15d-978403bca4d9\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " Sep 18 21:33:53.211548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211370 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-dshm\") pod \"4658f41a-d017-41c3-a15d-978403bca4d9\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " Sep 18 21:33:53.211548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211398 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-model-cache\") pod \"4658f41a-d017-41c3-a15d-978403bca4d9\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " Sep 18 21:33:53.211548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211447 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4658f41a-d017-41c3-a15d-978403bca4d9-tls-certs\") pod \"4658f41a-d017-41c3-a15d-978403bca4d9\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " Sep 18 21:33:53.211548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211509 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrxsr\" (UniqueName: \"kubernetes.io/projected/4658f41a-d017-41c3-a15d-978403bca4d9-kube-api-access-lrxsr\") pod \"4658f41a-d017-41c3-a15d-978403bca4d9\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " Sep 18 21:33:53.211548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211549 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-home\") pod \"4658f41a-d017-41c3-a15d-978403bca4d9\" (UID: \"4658f41a-d017-41c3-a15d-978403bca4d9\") " Sep 18 21:33:53.211839 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211682 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "4658f41a-d017-41c3-a15d-978403bca4d9" (UID: "4658f41a-d017-41c3-a15d-978403bca4d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:53.211921 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211892 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4658f41a-d017-41c3-a15d-978403bca4d9" (UID: "4658f41a-d017-41c3-a15d-978403bca4d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:53.211921 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211906 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-home" (OuterVolumeSpecName: "home") pod "4658f41a-d017-41c3-a15d-978403bca4d9" (UID: "4658f41a-d017-41c3-a15d-978403bca4d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:53.212079 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.211912 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:53.214099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.214066 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4658f41a-d017-41c3-a15d-978403bca4d9-kube-api-access-lrxsr" (OuterVolumeSpecName: "kube-api-access-lrxsr") pod "4658f41a-d017-41c3-a15d-978403bca4d9" (UID: "4658f41a-d017-41c3-a15d-978403bca4d9"). InnerVolumeSpecName "kube-api-access-lrxsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:33:53.214534 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.214510 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4658f41a-d017-41c3-a15d-978403bca4d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4658f41a-d017-41c3-a15d-978403bca4d9" (UID: "4658f41a-d017-41c3-a15d-978403bca4d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:33:53.214665 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.214637 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-dshm" (OuterVolumeSpecName: "dshm") pod "4658f41a-d017-41c3-a15d-978403bca4d9" (UID: "4658f41a-d017-41c3-a15d-978403bca4d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:33:53.313009 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.312947 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:53.313009 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.312997 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:53.313009 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.313010 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4658f41a-d017-41c3-a15d-978403bca4d9-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:53.313293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.313024 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrxsr\" (UniqueName: \"kubernetes.io/projected/4658f41a-d017-41c3-a15d-978403bca4d9-kube-api-access-lrxsr\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:53.313293 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.313035 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4658f41a-d017-41c3-a15d-978403bca4d9-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:33:53.844677 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.844635 2577 generic.go:356] "Generic (PLEG): container finished" podID="4658f41a-d017-41c3-a15d-978403bca4d9" containerID="cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc" exitCode=0 Sep 18 21:33:53.844871 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.844697 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" Sep 18 21:33:53.844871 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.844716 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" event={"ID":"4658f41a-d017-41c3-a15d-978403bca4d9","Type":"ContainerDied","Data":"cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc"} Sep 18 21:33:53.844871 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.844759 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7" event={"ID":"4658f41a-d017-41c3-a15d-978403bca4d9","Type":"ContainerDied","Data":"a90fc4675f71c2f62f3984c626d3a277b3921d16ac8ed8b68da83bfc8341aee2"} Sep 18 21:33:53.844871 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.844776 2577 scope.go:117] "RemoveContainer" containerID="cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc" Sep 18 21:33:53.854005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.853988 2577 scope.go:117] "RemoveContainer" containerID="cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc" Sep 18 21:33:53.854306 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:33:53.854283 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc\": container with ID starting with cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc not found: ID does not exist" containerID="cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc" Sep 18 21:33:53.854356 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.854316 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc"} err="failed to get container status \"cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc\": rpc error: code = NotFound desc = could not find container \"cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc\": container with ID starting with cff5acc0832b924e65f68f031cfa257d684c208958af69d9935c49e309204ddc not found: ID does not exist" Sep 18 21:33:53.869588 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.869555 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7"] Sep 18 21:33:53.876171 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:53.876138 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/sub-test-model-cad5cf-kserve-6d7c485b8c-24gn7"] Sep 18 21:33:55.360326 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:33:55.360289 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4658f41a-d017-41c3-a15d-978403bca4d9" path="/var/lib/kubelet/pods/4658f41a-d017-41c3-a15d-978403bca4d9/volumes" Sep 18 21:34:01.551324 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551288 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4"] Sep 18 21:34:01.551773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551714 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4658f41a-d017-41c3-a15d-978403bca4d9" containerName="main" Sep 18 21:34:01.551773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551726 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4658f41a-d017-41c3-a15d-978403bca4d9" containerName="main" Sep 18 21:34:01.551773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551740 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" Sep 18 21:34:01.551773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551746 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" Sep 18 21:34:01.551773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551753 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" Sep 18 21:34:01.551773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551759 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" Sep 18 21:34:01.551773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551771 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" Sep 18 21:34:01.551773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551776 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551785 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551792 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551798 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4786713e-acfd-4fa6-a80b-e0d053295437" containerName="revoke-keys" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551803 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4786713e-acfd-4fa6-a80b-e0d053295437" containerName="revoke-keys" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551876 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551883 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551890 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4786713e-acfd-4fa6-a80b-e0d053295437" containerName="revoke-keys" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551896 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4658f41a-d017-41c3-a15d-978403bca4d9" containerName="main" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551902 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551928 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" Sep 18 21:34:01.552059 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.551935 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c20cb642-c603-4d97-8e6f-0ec218d3fa0b" containerName="istio-proxy" Sep 18 21:34:01.556470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.556449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.572092 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.572058 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cad5cf\"/\"kube-root-ca.crt\"" Sep 18 21:34:01.572592 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.572564 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cad5cf\"/\"openshift-service-ca.crt\"" Sep 18 21:34:01.572757 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.572564 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cad5cf\"/\"default-dockercfg-8qzpg\"" Sep 18 21:34:01.572757 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.572571 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cad5cf\"/\"rate-test-model-cad5cf-kserve-self-signed-certs\"" Sep 18 21:34:01.589883 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.589846 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4"] Sep 18 21:34:01.699744 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.699700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgrf\" (UniqueName: \"kubernetes.io/projected/8a2751ff-e588-43c7-ad36-f6da902cf742-kube-api-access-sjgrf\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.700031 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.699763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2751ff-e588-43c7-ad36-f6da902cf742-tls-certs\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.700031 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.699827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-home\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.700031 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.699926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-tmp-dir\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.700031 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.699989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-dshm\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.700205 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.700083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-model-cache\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.801273 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.801236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2751ff-e588-43c7-ad36-f6da902cf742-tls-certs\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.801638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.801293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-home\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.801638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.801339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-tmp-dir\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.801802 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.801779 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-tmp-dir\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.801867 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.801818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-dshm\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.801936 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.801884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-model-cache\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.801936 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.801886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-home\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.802092 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.802007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgrf\" (UniqueName: \"kubernetes.io/projected/8a2751ff-e588-43c7-ad36-f6da902cf742-kube-api-access-sjgrf\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.802153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.802122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-model-cache\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.804650 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.804626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2751ff-e588-43c7-ad36-f6da902cf742-tls-certs\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.806417 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.806386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-dshm\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.815927 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.815894 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgrf\" (UniqueName: \"kubernetes.io/projected/8a2751ff-e588-43c7-ad36-f6da902cf742-kube-api-access-sjgrf\") pod \"rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:01.865789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:01.865740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:02.161159 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:34:02.161122 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a2751ff_e588_43c7_ad36_f6da902cf742.slice/crio-f2ce1017e0aaa99203fd6d08d62b3559dc5e2731046659053b8b915767f03bcf WatchSource:0}: Error finding container f2ce1017e0aaa99203fd6d08d62b3559dc5e2731046659053b8b915767f03bcf: Status 404 returned error can't find the container with id f2ce1017e0aaa99203fd6d08d62b3559dc5e2731046659053b8b915767f03bcf Sep 18 21:34:02.161612 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:02.161580 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4"] Sep 18 21:34:02.911682 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:02.911643 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" event={"ID":"8a2751ff-e588-43c7-ad36-f6da902cf742","Type":"ContainerStarted","Data":"ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1"} Sep 18 21:34:02.911682 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:02.911689 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" event={"ID":"8a2751ff-e588-43c7-ad36-f6da902cf742","Type":"ContainerStarted","Data":"f2ce1017e0aaa99203fd6d08d62b3559dc5e2731046659053b8b915767f03bcf"} Sep 18 21:34:02.961548 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:02.961490 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" podStartSLOduration=1.961472685 podStartE2EDuration="1.961472685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:34:02.951000847 +0000 UTC m=+1671.880746717" watchObservedRunningTime="2026-09-18 21:34:02.961472685 +0000 UTC m=+1671.891218557" Sep 18 21:34:11.866900 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:11.866859 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:11.867386 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:11.867174 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:11.880261 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:11.880223 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:34:11.985880 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:34:11.985849 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:35:13.655657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:13.655626 2577 scope.go:117] "RemoveContainer" containerID="ee9b046b8b0b7d922386e94aeffe0e1c7241b76d5d71d35331a8a2c2bfc9acca" Sep 18 21:35:33.661625 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:33.661580 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4"] Sep 18 21:35:35.351539 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.351503 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz"] Sep 18 21:35:35.351991 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.351965 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" Sep 18 21:35:35.351991 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.351984 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deb7beb-dcfa-4cbf-904b-6816e91c3183" containerName="istio-proxy" Sep 18 21:35:35.352099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.352011 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" Sep 18 21:35:35.352099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.352017 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" Sep 18 21:35:35.352099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.352080 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee10e005-3476-4a74-af72-004006b8c085" containerName="istio-proxy" Sep 18 21:35:35.355387 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.355364 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.358131 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.358108 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5888e6-openshift-default-dockercfg-sztpw\"" Sep 18 21:35:35.367761 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.367731 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz"] Sep 18 21:35:35.450564 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.450526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istiod-ca-cert\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.450760 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.450729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-socket\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.450835 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.450765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-credential-socket\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.450835 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.450798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-podinfo\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.450942 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.450883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-data\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.451022 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.450939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-certs\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.451080 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.451018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-envoy\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.451080 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.451060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-token\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.451180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.451123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452ms\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-kube-api-access-452ms\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.551673 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-socket\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.551673 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-credential-socket\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.551904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-podinfo\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.551904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-data\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.551904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-certs\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.551904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-envoy\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.551904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551837 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-token\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.551904 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452ms\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-kube-api-access-452ms\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.552246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.551926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istiod-ca-cert\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.552246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.552123 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-socket\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.552246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.552213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-data\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.552426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.552272 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-certs\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.552426 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.552380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-credential-socket\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.552609 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.552580 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istiod-ca-cert\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.554147 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.554125 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-envoy\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.554229 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.554207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-podinfo\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.561164 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.561142 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-token\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.561317 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.561302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-452ms\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-kube-api-access-452ms\") pod \"e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.668173 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.668088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:35.947602 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.947574 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz"] Sep 18 21:35:35.949799 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:35:35.949769 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a34c13_ead1_4fb1_8552_8aeed5b3cec0.slice/crio-4e8db9181f40fb8cc36d2873b9361f223ab90ac8dbdd7e94a4e8fd8a1b47880b WatchSource:0}: Error finding container 4e8db9181f40fb8cc36d2873b9361f223ab90ac8dbdd7e94a4e8fd8a1b47880b: Status 404 returned error can't find the container with id 4e8db9181f40fb8cc36d2873b9361f223ab90ac8dbdd7e94a4e8fd8a1b47880b Sep 18 21:35:35.952158 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.952128 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:35:35.952241 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.952193 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:35:35.952241 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:35.952224 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:35:36.530148 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:36.530109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" event={"ID":"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0","Type":"ContainerStarted","Data":"8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d"} Sep 18 21:35:36.530634 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:36.530176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" event={"ID":"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0","Type":"ContainerStarted","Data":"4e8db9181f40fb8cc36d2873b9361f223ab90ac8dbdd7e94a4e8fd8a1b47880b"} Sep 18 21:35:36.554704 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:36.554650 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" podStartSLOduration=1.5546355109999999 podStartE2EDuration="1.554635511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:35:36.55272338 +0000 UTC m=+1765.482469252" watchObservedRunningTime="2026-09-18 21:35:36.554635511 +0000 UTC m=+1765.484381447" Sep 18 21:35:36.669255 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:36.669209 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:37.673894 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:37.673862 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:38.545703 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:38.545662 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:38.546560 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:38.546541 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:48.680635 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:48.680565 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" podUID="8a2751ff-e588-43c7-ad36-f6da902cf742" containerName="main" containerID="cri-o://ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1" gracePeriod=45 Sep 18 21:35:48.991748 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:48.991723 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:35:49.095576 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.095541 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-model-cache\") pod \"8a2751ff-e588-43c7-ad36-f6da902cf742\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " Sep 18 21:35:49.095780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.095606 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-tmp-dir\") pod \"8a2751ff-e588-43c7-ad36-f6da902cf742\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " Sep 18 21:35:49.095780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.095660 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-dshm\") pod \"8a2751ff-e588-43c7-ad36-f6da902cf742\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " Sep 18 21:35:49.095780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.095704 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2751ff-e588-43c7-ad36-f6da902cf742-tls-certs\") pod \"8a2751ff-e588-43c7-ad36-f6da902cf742\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " Sep 18 21:35:49.095780 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.095739 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-home\") pod \"8a2751ff-e588-43c7-ad36-f6da902cf742\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " Sep 18 21:35:49.096016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.095828 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjgrf\" (UniqueName: \"kubernetes.io/projected/8a2751ff-e588-43c7-ad36-f6da902cf742-kube-api-access-sjgrf\") pod \"8a2751ff-e588-43c7-ad36-f6da902cf742\" (UID: \"8a2751ff-e588-43c7-ad36-f6da902cf742\") " Sep 18 21:35:49.096016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.095836 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a2751ff-e588-43c7-ad36-f6da902cf742" (UID: "8a2751ff-e588-43c7-ad36-f6da902cf742"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:35:49.096016 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.095881 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a2751ff-e588-43c7-ad36-f6da902cf742" (UID: "8a2751ff-e588-43c7-ad36-f6da902cf742"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:35:49.096165 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.096067 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-home" (OuterVolumeSpecName: "home") pod "8a2751ff-e588-43c7-ad36-f6da902cf742" (UID: "8a2751ff-e588-43c7-ad36-f6da902cf742"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:35:49.096246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.096214 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:35:49.096246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.096234 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:35:49.096246 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.096245 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:35:49.097978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.097924 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-dshm" (OuterVolumeSpecName: "dshm") pod "8a2751ff-e588-43c7-ad36-f6da902cf742" (UID: "8a2751ff-e588-43c7-ad36-f6da902cf742"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:35:49.097978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.097948 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2751ff-e588-43c7-ad36-f6da902cf742-kube-api-access-sjgrf" (OuterVolumeSpecName: "kube-api-access-sjgrf") pod "8a2751ff-e588-43c7-ad36-f6da902cf742" (UID: "8a2751ff-e588-43c7-ad36-f6da902cf742"). InnerVolumeSpecName "kube-api-access-sjgrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:35:49.098133 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.097996 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2751ff-e588-43c7-ad36-f6da902cf742-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a2751ff-e588-43c7-ad36-f6da902cf742" (UID: "8a2751ff-e588-43c7-ad36-f6da902cf742"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:35:49.197304 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.197263 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a2751ff-e588-43c7-ad36-f6da902cf742-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:35:49.197304 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.197298 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a2751ff-e588-43c7-ad36-f6da902cf742-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:35:49.197510 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.197312 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgrf\" (UniqueName: \"kubernetes.io/projected/8a2751ff-e588-43c7-ad36-f6da902cf742-kube-api-access-sjgrf\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:35:49.616317 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.616281 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a2751ff-e588-43c7-ad36-f6da902cf742" containerID="ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1" exitCode=0 Sep 18 21:35:49.616524 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.616327 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" Sep 18 21:35:49.616524 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.616370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" event={"ID":"8a2751ff-e588-43c7-ad36-f6da902cf742","Type":"ContainerDied","Data":"ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1"} Sep 18 21:35:49.616524 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.616421 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4" event={"ID":"8a2751ff-e588-43c7-ad36-f6da902cf742","Type":"ContainerDied","Data":"f2ce1017e0aaa99203fd6d08d62b3559dc5e2731046659053b8b915767f03bcf"} Sep 18 21:35:49.616524 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.616441 2577 scope.go:117] "RemoveContainer" containerID="ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1" Sep 18 21:35:49.625878 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.625861 2577 scope.go:117] "RemoveContainer" containerID="ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1" Sep 18 21:35:49.626186 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:35:49.626164 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1\": container with ID starting with ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1 not found: ID does not exist" containerID="ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1" Sep 18 21:35:49.626243 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.626196 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1"} err="failed to get container status \"ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1\": rpc error: code = NotFound desc = could not find container \"ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1\": container with ID starting with ce37e2f7d4245a4261c3d79ee986286bbf2b57c1fec411ac288735b38e1a77b1 not found: ID does not exist" Sep 18 21:35:49.638645 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.638604 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4"] Sep 18 21:35:49.655978 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:49.654557 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cad5cf/rate-test-model-cad5cf-kserve-5ff7b5db99-4hks4"] Sep 18 21:35:51.360856 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:51.360818 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2751ff-e588-43c7-ad36-f6da902cf742" path="/var/lib/kubelet/pods/8a2751ff-e588-43c7-ad36-f6da902cf742/volumes" Sep 18 21:35:54.587249 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:54.587144 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 18 21:35:54.587249 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:54.587216 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 21:35:55.661104 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:55.661064 2577 generic.go:356] "Generic (PLEG): container finished" podID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerID="8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d" exitCode=0 Sep 18 21:35:55.661547 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:55.661137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" event={"ID":"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0","Type":"ContainerDied","Data":"8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d"} Sep 18 21:35:55.661628 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:55.661566 2577 scope.go:117] "RemoveContainer" containerID="8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d" Sep 18 21:35:55.662793 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:55.662764 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:35:55.662857 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:55.662837 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:35:55.662894 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:55.662866 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:35:56.676528 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:56.676480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" event={"ID":"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0","Type":"ContainerStarted","Data":"d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f"} Sep 18 21:35:57.668899 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:57.668861 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:35:57.669142 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:57.669116 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 18 21:35:57.669195 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:57.669171 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 18 21:35:58.668989 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:58.668923 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 18 21:35:58.669484 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:58.669020 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 18 21:35:59.668552 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:59.668515 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 18 21:35:59.668745 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:35:59.668576 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 18 21:36:00.668578 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:00.668542 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 18 21:36:00.669001 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:00.668596 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 18 21:36:01.669076 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:01.669036 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 18 21:36:01.669558 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:01.669104 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 18 21:36:02.672495 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:02.672462 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:36:02.672973 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:02.672768 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:36:02.673387 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:02.673373 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:36:04.429887 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.429854 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z"] Sep 18 21:36:04.430296 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.430288 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a2751ff-e588-43c7-ad36-f6da902cf742" containerName="main" Sep 18 21:36:04.430342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.430299 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a2751ff-e588-43c7-ad36-f6da902cf742" containerName="main" Sep 18 21:36:04.430398 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.430387 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a2751ff-e588-43c7-ad36-f6da902cf742" containerName="main" Sep 18 21:36:04.432457 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.432439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.434854 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.434831 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-deac7a-openshift-default-dockercfg-zw646\"" Sep 18 21:36:04.450209 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.450185 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z"] Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.554511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-token\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.554587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-envoy\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.554638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-credential-socket\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.554692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-data\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.554726 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-socket\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.554788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930099c3-dddd-4873-82ee-7f5de6a1a64d-istiod-ca-cert\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.554898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-certs\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.554997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bszpf\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-kube-api-access-bszpf\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.555220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.555069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-podinfo\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.655992 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.655929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-podinfo\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656160 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-token\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656160 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-envoy\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656160 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-credential-socket\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656160 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-data\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656160 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-socket\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656431 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930099c3-dddd-4873-82ee-7f5de6a1a64d-istiod-ca-cert\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656431 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-certs\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656431 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszpf\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-kube-api-access-bszpf\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656623 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-data\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656615 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-credential-socket\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656774 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-certs\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.656947 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.656918 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-socket\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.657103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.657081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930099c3-dddd-4873-82ee-7f5de6a1a64d-istiod-ca-cert\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.658382 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.658360 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-envoy\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.658787 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.658768 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-podinfo\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.663886 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.663861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-token\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.664068 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.664051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bszpf\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-kube-api-access-bszpf\") pod \"e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:04.745286 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:04.745199 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:05.029223 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:05.029187 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z"] Sep 18 21:36:05.029665 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:36:05.029638 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930099c3_dddd_4873_82ee_7f5de6a1a64d.slice/crio-9c49477246a842f300cb24861f1ede06a51b69d5545005d01572b992b278e1cb WatchSource:0}: Error finding container 9c49477246a842f300cb24861f1ede06a51b69d5545005d01572b992b278e1cb: Status 404 returned error can't find the container with id 9c49477246a842f300cb24861f1ede06a51b69d5545005d01572b992b278e1cb Sep 18 21:36:05.032610 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:05.032541 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:36:05.032675 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:05.032651 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:36:05.032729 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:05.032690 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:36:05.739701 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:05.739657 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" event={"ID":"930099c3-dddd-4873-82ee-7f5de6a1a64d","Type":"ContainerStarted","Data":"8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98"} Sep 18 21:36:05.740197 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:05.739754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" event={"ID":"930099c3-dddd-4873-82ee-7f5de6a1a64d","Type":"ContainerStarted","Data":"9c49477246a842f300cb24861f1ede06a51b69d5545005d01572b992b278e1cb"} Sep 18 21:36:05.745983 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:05.745946 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:05.774210 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:05.774165 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" podStartSLOduration=1.774151469 podStartE2EDuration="1.774151469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:36:05.771044862 +0000 UTC m=+1794.700790734" watchObservedRunningTime="2026-09-18 21:36:05.774151469 +0000 UTC m=+1794.703897342" Sep 18 21:36:06.751544 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:06.751516 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:07.752518 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:07.752483 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:07.753446 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:07.753430 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:13.503305 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:13.503279 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:36:13.509394 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:13.509374 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:36:13.682103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:13.682070 2577 scope.go:117] "RemoveContainer" containerID="c460146bccbaf9de009c52266d0edfc192cef8388097a2bc7192ea93b4c5c369" Sep 18 21:36:13.693334 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:13.693292 2577 scope.go:117] "RemoveContainer" containerID="625f89daf8f395cfa6723815b8aac282adf817b975c647406e5b69a14202ac08" Sep 18 21:36:13.701573 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:13.701553 2577 scope.go:117] "RemoveContainer" containerID="e24a6ff3e8fb600fcdac0ce4b5807b3dab47ff222323089ceba9002c5affd2ed" Sep 18 21:36:26.883821 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:26.883735 2577 generic.go:356] "Generic (PLEG): container finished" podID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerID="8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98" exitCode=0 Sep 18 21:36:26.883821 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:26.883807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" event={"ID":"930099c3-dddd-4873-82ee-7f5de6a1a64d","Type":"ContainerDied","Data":"8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98"} Sep 18 21:36:26.884291 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:26.884260 2577 scope.go:117] "RemoveContainer" containerID="8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98" Sep 18 21:36:26.885374 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:26.885349 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:36:26.885432 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:26.885418 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:36:26.885468 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:26.885457 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:36:27.891763 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:27.891731 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" event={"ID":"930099c3-dddd-4873-82ee-7f5de6a1a64d","Type":"ContainerStarted","Data":"7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590"} Sep 18 21:36:28.746151 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:28.746117 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:28.746526 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:28.746447 2577 patch_prober.go:28] interesting pod/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 18 21:36:28.746693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:28.746667 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 18 21:36:29.746110 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:29.746070 2577 patch_prober.go:28] interesting pod/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 18 21:36:29.746707 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:29.746672 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 18 21:36:30.746105 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:30.745665 2577 patch_prober.go:28] interesting pod/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 18 21:36:30.746105 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:30.745727 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 18 21:36:31.746537 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:31.746487 2577 patch_prober.go:28] interesting pod/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 18 21:36:31.747054 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:31.746582 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 18 21:36:32.745893 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:32.745847 2577 patch_prober.go:28] interesting pod/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 18 21:36:32.746102 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:32.745918 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 18 21:36:33.749540 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:33.749508 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:33.749946 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:33.749857 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:33.750659 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:33.750640 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:36:50.595510 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.595430 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q"] Sep 18 21:36:50.598342 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.598322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.601026 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.601003 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5888e6\"/\"openshift-service-ca.crt\"" Sep 18 21:36:50.601712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.601687 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5888e6\"/\"ipp-route-5888e6-kserve-self-signed-certs\"" Sep 18 21:36:50.601712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.601710 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5888e6\"/\"default-dockercfg-h894z\"" Sep 18 21:36:50.601893 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.601791 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5888e6\"/\"kube-root-ca.crt\"" Sep 18 21:36:50.609860 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.609834 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q"] Sep 18 21:36:50.641554 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.641522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24574153-8abb-4119-9441-77b843d3573f-tls-certs\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.641716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.641576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-model-cache\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.641716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.641654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-home\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.641716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.641684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-tmp-dir\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.641830 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.641716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8htr\" (UniqueName: \"kubernetes.io/projected/24574153-8abb-4119-9441-77b843d3573f-kube-api-access-c8htr\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.641830 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.641812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-dshm\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.742638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.742604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-model-cache\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.742829 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.742674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-home\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.742829 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.742704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-tmp-dir\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.742829 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.742750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8htr\" (UniqueName: \"kubernetes.io/projected/24574153-8abb-4119-9441-77b843d3573f-kube-api-access-c8htr\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.742829 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.742815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-dshm\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.743124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.742854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24574153-8abb-4119-9441-77b843d3573f-tls-certs\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.743124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.743072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-model-cache\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.743233 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.743180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-tmp-dir\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.743233 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.743216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-home\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.745206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.745184 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-dshm\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.745366 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.745349 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24574153-8abb-4119-9441-77b843d3573f-tls-certs\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.751087 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.751068 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8htr\" (UniqueName: \"kubernetes.io/projected/24574153-8abb-4119-9441-77b843d3573f-kube-api-access-c8htr\") pod \"ipp-route-5888e6-kserve-cf9469b5f-bpz2q\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:50.909824 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:50.909746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:36:51.186307 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:51.186280 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q"] Sep 18 21:36:51.187536 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:36:51.187507 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24574153_8abb_4119_9441_77b843d3573f.slice/crio-4e1e0bf62524badaf9c2bc693f243b50bb8cf0feacf3ee45c0892342babeb573 WatchSource:0}: Error finding container 4e1e0bf62524badaf9c2bc693f243b50bb8cf0feacf3ee45c0892342babeb573: Status 404 returned error can't find the container with id 4e1e0bf62524badaf9c2bc693f243b50bb8cf0feacf3ee45c0892342babeb573 Sep 18 21:36:52.049113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:52.049071 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" event={"ID":"24574153-8abb-4119-9441-77b843d3573f","Type":"ContainerStarted","Data":"6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087"} Sep 18 21:36:52.049566 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:52.049121 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" event={"ID":"24574153-8abb-4119-9441-77b843d3573f","Type":"ContainerStarted","Data":"4e1e0bf62524badaf9c2bc693f243b50bb8cf0feacf3ee45c0892342babeb573"} Sep 18 21:36:52.068268 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:36:52.068212 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" podStartSLOduration=2.0681974260000002 podStartE2EDuration="2.068197426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:36:52.066583825 +0000 UTC m=+1840.996329698" watchObservedRunningTime="2026-09-18 21:36:52.068197426 +0000 UTC m=+1840.997943299" Sep 18 21:37:00.910848 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:00.910811 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:37:00.911407 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:00.910862 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:37:00.923063 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:00.923039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:37:01.118995 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:01.118949 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:37:13.735152 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:13.735119 2577 scope.go:117] "RemoveContainer" containerID="fbb0235fc59ce57e42db8a1d1b446c3ebe40f6d32bfeb09e30fba5792b512d37" Sep 18 21:37:19.902388 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:19.902352 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8"] Sep 18 21:37:19.905634 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:19.905609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:19.908177 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:19.908160 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-b3d150-openshift-default-dockercfg-m6dpf\"" Sep 18 21:37:19.924403 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:19.924374 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8"] Sep 18 21:37:20.053669 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.053631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkj6\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-kube-api-access-klkj6\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.053669 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.053682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-podinfo\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.053909 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.053732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5150ffbc-b2f7-4a65-94e2-beea89b65129-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.053909 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.053749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-token\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.053909 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.053873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-socket\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.054045 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.053914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-envoy\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.054045 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.054003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-credential-socket\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.054045 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.054032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-certs\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.054153 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.054055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-data\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.155713 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klkj6\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-kube-api-access-klkj6\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.155713 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-podinfo\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.155713 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5150ffbc-b2f7-4a65-94e2-beea89b65129-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.155713 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-token\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156108 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-socket\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156108 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-envoy\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156108 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-credential-socket\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156108 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-certs\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156108 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.155867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-data\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156374 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.156337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-socket\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156433 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.156396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-credential-socket\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156501 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.156445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-data\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156561 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.156535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-certs\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.156658 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.156638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5150ffbc-b2f7-4a65-94e2-beea89b65129-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.158281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.158258 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-envoy\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.158429 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.158411 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-podinfo\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.169657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.169591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkj6\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-kube-api-access-klkj6\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.170830 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.170801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-token\") pod \"e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.215545 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.215511 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:20.496741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.496709 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8"] Sep 18 21:37:20.497456 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:37:20.497429 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5150ffbc_b2f7_4a65_94e2_beea89b65129.slice/crio-927ebcfea53b859c2fde221ae4380f66c7e448e81e72f15ecab98ed03628aecf WatchSource:0}: Error finding container 927ebcfea53b859c2fde221ae4380f66c7e448e81e72f15ecab98ed03628aecf: Status 404 returned error can't find the container with id 927ebcfea53b859c2fde221ae4380f66c7e448e81e72f15ecab98ed03628aecf Sep 18 21:37:20.499876 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.499841 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:37:20.499949 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.499913 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:37:20.499949 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:20.499941 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:37:21.244467 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:21.244427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" event={"ID":"5150ffbc-b2f7-4a65-94e2-beea89b65129","Type":"ContainerStarted","Data":"27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65"} Sep 18 21:37:21.244902 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:21.244496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" event={"ID":"5150ffbc-b2f7-4a65-94e2-beea89b65129","Type":"ContainerStarted","Data":"927ebcfea53b859c2fde221ae4380f66c7e448e81e72f15ecab98ed03628aecf"} Sep 18 21:37:21.270775 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:21.270724 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" podStartSLOduration=2.270710345 podStartE2EDuration="2.270710345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:37:21.267175808 +0000 UTC m=+1870.196921685" watchObservedRunningTime="2026-09-18 21:37:21.270710345 +0000 UTC m=+1870.200456219" Sep 18 21:37:22.216726 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:22.216683 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:22.222614 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:22.222584 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:22.252477 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:22.252444 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:22.253387 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:22.253360 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:43.417718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:43.417621 2577 generic.go:356] "Generic (PLEG): container finished" podID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerID="27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65" exitCode=0 Sep 18 21:37:43.417718 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:43.417704 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" event={"ID":"5150ffbc-b2f7-4a65-94e2-beea89b65129","Type":"ContainerDied","Data":"27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65"} Sep 18 21:37:43.418166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:43.418101 2577 scope.go:117] "RemoveContainer" containerID="27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65" Sep 18 21:37:43.419187 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:43.419162 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:37:43.419255 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:43.419223 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:37:43.419292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:43.419252 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 18 21:37:44.427533 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:44.427491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" event={"ID":"5150ffbc-b2f7-4a65-94e2-beea89b65129","Type":"ContainerStarted","Data":"c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd"} Sep 18 21:37:45.216545 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:45.216511 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:45.216763 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:45.216738 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Sep 18 21:37:45.216824 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:45.216806 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Sep 18 21:37:46.216015 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:46.215978 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Sep 18 21:37:46.216392 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:46.216033 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Sep 18 21:37:47.216453 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:47.216409 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Sep 18 21:37:47.216884 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:47.216474 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Sep 18 21:37:48.216349 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:48.216315 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Sep 18 21:37:48.216527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:48.216368 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Sep 18 21:37:49.216728 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:49.216692 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Sep 18 21:37:49.217181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:49.216755 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Sep 18 21:37:50.219852 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:50.219816 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:50.220280 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:50.220163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:50.220745 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:50.220730 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:37:53.985588 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:53.985545 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2"] Sep 18 21:37:53.988465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:53.988443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" Sep 18 21:37:53.990880 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:53.990858 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 21:37:53.991196 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:53.991180 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-52bhz\"" Sep 18 21:37:53.999474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:53.999448 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2"] Sep 18 21:37:54.121535 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:54.121495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6cb65-60c4-4eab-9692-0a1db95ed057-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2\" (UID: \"5ef6cb65-60c4-4eab-9692-0a1db95ed057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" Sep 18 21:37:54.222614 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:54.222577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6cb65-60c4-4eab-9692-0a1db95ed057-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2\" (UID: \"5ef6cb65-60c4-4eab-9692-0a1db95ed057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" Sep 18 21:37:54.223264 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:54.223241 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6cb65-60c4-4eab-9692-0a1db95ed057-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2\" (UID: \"5ef6cb65-60c4-4eab-9692-0a1db95ed057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" Sep 18 21:37:54.300436 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:54.300354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" Sep 18 21:37:54.576344 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:54.576319 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2"] Sep 18 21:37:54.577460 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:37:54.577436 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef6cb65_60c4_4eab_9692_0a1db95ed057.slice/crio-d3f1193b3b534e784efb79a680bb68badeae9edb343294bcce50597bf435380e WatchSource:0}: Error finding container d3f1193b3b534e784efb79a680bb68badeae9edb343294bcce50597bf435380e: Status 404 returned error can't find the container with id d3f1193b3b534e784efb79a680bb68badeae9edb343294bcce50597bf435380e Sep 18 21:37:55.511060 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:55.510943 2577 generic.go:356] "Generic (PLEG): container finished" podID="5ef6cb65-60c4-4eab-9692-0a1db95ed057" containerID="ddfca5dad4e641b2272fcdb57de9c9d775fbe1003bb7f60b964e72699c3729bc" exitCode=0 Sep 18 21:37:55.511060 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:55.511033 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" event={"ID":"5ef6cb65-60c4-4eab-9692-0a1db95ed057","Type":"ContainerDied","Data":"ddfca5dad4e641b2272fcdb57de9c9d775fbe1003bb7f60b964e72699c3729bc"} Sep 18 21:37:55.511449 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:55.511080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" event={"ID":"5ef6cb65-60c4-4eab-9692-0a1db95ed057","Type":"ContainerStarted","Data":"d3f1193b3b534e784efb79a680bb68badeae9edb343294bcce50597bf435380e"} Sep 18 21:37:56.657690 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:56.657663 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" Sep 18 21:37:56.751597 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:56.751567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6cb65-60c4-4eab-9692-0a1db95ed057-maas-api-service-ca\") pod \"5ef6cb65-60c4-4eab-9692-0a1db95ed057\" (UID: \"5ef6cb65-60c4-4eab-9692-0a1db95ed057\") " Sep 18 21:37:56.752024 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:56.751991 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef6cb65-60c4-4eab-9692-0a1db95ed057-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ef6cb65-60c4-4eab-9692-0a1db95ed057" (UID: "5ef6cb65-60c4-4eab-9692-0a1db95ed057"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:37:56.853202 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:56.853167 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef6cb65-60c4-4eab-9692-0a1db95ed057-maas-api-service-ca\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:37:57.524159 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:57.524132 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" Sep 18 21:37:57.524331 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:57.524134 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2" event={"ID":"5ef6cb65-60c4-4eab-9692-0a1db95ed057","Type":"ContainerDied","Data":"d3f1193b3b534e784efb79a680bb68badeae9edb343294bcce50597bf435380e"} Sep 18 21:37:57.524331 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:37:57.524253 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3f1193b3b534e784efb79a680bb68badeae9edb343294bcce50597bf435380e" Sep 18 21:38:05.471882 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:05.471841 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2"] Sep 18 21:38:05.477713 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:05.477686 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b3d150-sr6c2"] Sep 18 21:38:07.361145 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:07.361109 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef6cb65-60c4-4eab-9692-0a1db95ed057" path="/var/lib/kubelet/pods/5ef6cb65-60c4-4eab-9692-0a1db95ed057/volumes" Sep 18 21:38:12.759868 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:12.759827 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8"] Sep 18 21:38:12.760424 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:12.760125 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" containerID="cri-o://c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd" gracePeriod=30 Sep 18 21:38:18.016770 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.016743 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:38:18.084578 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084538 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5150ffbc-b2f7-4a65-94e2-beea89b65129-istiod-ca-cert\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.084789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084604 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-podinfo\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.084789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084693 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-socket\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.084789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084717 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-data\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.084789 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084743 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klkj6\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-kube-api-access-klkj6\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.085168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084792 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-certs\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.085168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084820 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-token\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.085168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084856 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-credential-socket\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.085168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084899 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-envoy\") pod \"5150ffbc-b2f7-4a65-94e2-beea89b65129\" (UID: \"5150ffbc-b2f7-4a65-94e2-beea89b65129\") " Sep 18 21:38:18.085168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.084928 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5150ffbc-b2f7-4a65-94e2-beea89b65129-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:38:18.085168 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.085018 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:18.085458 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.085239 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:18.085458 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.085414 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5150ffbc-b2f7-4a65-94e2-beea89b65129-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.085458 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.085436 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.085458 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.085451 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.085648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.085188 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:18.085648 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.085502 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-data" (OuterVolumeSpecName: "istio-data") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:18.087223 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.087171 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-token" (OuterVolumeSpecName: "istio-token") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:38:18.087573 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.087544 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-kube-api-access-klkj6" (OuterVolumeSpecName: "kube-api-access-klkj6") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "kube-api-access-klkj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:38:18.087693 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.087635 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:38:18.087756 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.087684 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5150ffbc-b2f7-4a65-94e2-beea89b65129" (UID: "5150ffbc-b2f7-4a65-94e2-beea89b65129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:18.186498 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.186464 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.186498 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.186494 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.186498 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.186504 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klkj6\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-kube-api-access-klkj6\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.186750 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.186515 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.186750 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.186523 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.186750 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.186531 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5150ffbc-b2f7-4a65-94e2-beea89b65129-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:18.669652 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.669609 2577 generic.go:356] "Generic (PLEG): container finished" podID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerID="c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd" exitCode=0 Sep 18 21:38:18.669845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.669670 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" Sep 18 21:38:18.669845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.669690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" event={"ID":"5150ffbc-b2f7-4a65-94e2-beea89b65129","Type":"ContainerDied","Data":"c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd"} Sep 18 21:38:18.669845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.669749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8" event={"ID":"5150ffbc-b2f7-4a65-94e2-beea89b65129","Type":"ContainerDied","Data":"927ebcfea53b859c2fde221ae4380f66c7e448e81e72f15ecab98ed03628aecf"} Sep 18 21:38:18.669845 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.669773 2577 scope.go:117] "RemoveContainer" containerID="c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd" Sep 18 21:38:18.680435 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.680419 2577 scope.go:117] "RemoveContainer" containerID="27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65" Sep 18 21:38:18.689875 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.689853 2577 scope.go:117] "RemoveContainer" containerID="c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd" Sep 18 21:38:18.690217 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:38:18.690197 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd\": container with ID starting with c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd not found: ID does not exist" containerID="c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd" Sep 18 21:38:18.690276 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.690230 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd"} err="failed to get container status \"c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd\": rpc error: code = NotFound desc = could not find container \"c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd\": container with ID starting with c2a7be97988aa6b2402f880a957a0022c05bb5c9b88e9db06ced10dbb18bcecd not found: ID does not exist" Sep 18 21:38:18.690276 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.690250 2577 scope.go:117] "RemoveContainer" containerID="27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65" Sep 18 21:38:18.690506 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:38:18.690488 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65\": container with ID starting with 27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65 not found: ID does not exist" containerID="27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65" Sep 18 21:38:18.690555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.690509 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65"} err="failed to get container status \"27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65\": rpc error: code = NotFound desc = could not find container \"27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65\": container with ID starting with 27fbddc90f1a2623ce873c96e02c7e9d1c73b2751c139d606bb68de0912f4e65 not found: ID does not exist" Sep 18 21:38:18.703236 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.703204 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8"] Sep 18 21:38:18.705987 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:18.705943 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b3d150-openshift-default-68cdd48598-gmwk8"] Sep 18 21:38:19.360457 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:19.360409 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" path="/var/lib/kubelet/pods/5150ffbc-b2f7-4a65-94e2-beea89b65129/volumes" Sep 18 21:38:38.318638 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:38.318598 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q"] Sep 18 21:38:53.342011 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.341947 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" podUID="24574153-8abb-4119-9441-77b843d3573f" containerName="main" containerID="cri-o://6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087" gracePeriod=45 Sep 18 21:38:53.600231 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.600161 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:38:53.767773 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.767725 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24574153-8abb-4119-9441-77b843d3573f-tls-certs\") pod \"24574153-8abb-4119-9441-77b843d3573f\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " Sep 18 21:38:53.768003 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.767833 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-model-cache\") pod \"24574153-8abb-4119-9441-77b843d3573f\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " Sep 18 21:38:53.768003 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.767918 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-dshm\") pod \"24574153-8abb-4119-9441-77b843d3573f\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " Sep 18 21:38:53.768003 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.767989 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-home\") pod \"24574153-8abb-4119-9441-77b843d3573f\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " Sep 18 21:38:53.768163 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.768026 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-tmp-dir\") pod \"24574153-8abb-4119-9441-77b843d3573f\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " Sep 18 21:38:53.768163 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.768061 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8htr\" (UniqueName: \"kubernetes.io/projected/24574153-8abb-4119-9441-77b843d3573f-kube-api-access-c8htr\") pod \"24574153-8abb-4119-9441-77b843d3573f\" (UID: \"24574153-8abb-4119-9441-77b843d3573f\") " Sep 18 21:38:53.768281 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.768160 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-model-cache" (OuterVolumeSpecName: "model-cache") pod "24574153-8abb-4119-9441-77b843d3573f" (UID: "24574153-8abb-4119-9441-77b843d3573f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:53.768341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.768310 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-home" (OuterVolumeSpecName: "home") pod "24574153-8abb-4119-9441-77b843d3573f" (UID: "24574153-8abb-4119-9441-77b843d3573f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:53.768390 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.768325 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24574153-8abb-4119-9441-77b843d3573f" (UID: "24574153-8abb-4119-9441-77b843d3573f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:53.768562 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.768544 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:53.768635 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.768567 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:53.768635 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.768583 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:53.770271 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.770243 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24574153-8abb-4119-9441-77b843d3573f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24574153-8abb-4119-9441-77b843d3573f" (UID: "24574153-8abb-4119-9441-77b843d3573f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:38:53.770271 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.770253 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-dshm" (OuterVolumeSpecName: "dshm") pod "24574153-8abb-4119-9441-77b843d3573f" (UID: "24574153-8abb-4119-9441-77b843d3573f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:38:53.770418 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.770295 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24574153-8abb-4119-9441-77b843d3573f-kube-api-access-c8htr" (OuterVolumeSpecName: "kube-api-access-c8htr") pod "24574153-8abb-4119-9441-77b843d3573f" (UID: "24574153-8abb-4119-9441-77b843d3573f"). InnerVolumeSpecName "kube-api-access-c8htr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:38:53.869478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.869379 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24574153-8abb-4119-9441-77b843d3573f-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:53.869478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.869424 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8htr\" (UniqueName: \"kubernetes.io/projected/24574153-8abb-4119-9441-77b843d3573f-kube-api-access-c8htr\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:53.869478 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.869436 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24574153-8abb-4119-9441-77b843d3573f-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:38:53.931785 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.931745 2577 generic.go:356] "Generic (PLEG): container finished" podID="24574153-8abb-4119-9441-77b843d3573f" containerID="6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087" exitCode=0 Sep 18 21:38:53.932017 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.931792 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" Sep 18 21:38:53.932017 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.931836 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" event={"ID":"24574153-8abb-4119-9441-77b843d3573f","Type":"ContainerDied","Data":"6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087"} Sep 18 21:38:53.932017 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.931886 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q" event={"ID":"24574153-8abb-4119-9441-77b843d3573f","Type":"ContainerDied","Data":"4e1e0bf62524badaf9c2bc693f243b50bb8cf0feacf3ee45c0892342babeb573"} Sep 18 21:38:53.932017 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.931904 2577 scope.go:117] "RemoveContainer" containerID="6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087" Sep 18 21:38:53.943647 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.943629 2577 scope.go:117] "RemoveContainer" containerID="6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087" Sep 18 21:38:53.943919 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:38:53.943899 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087\": container with ID starting with 6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087 not found: ID does not exist" containerID="6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087" Sep 18 21:38:53.944019 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.943931 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087"} err="failed to get container status \"6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087\": rpc error: code = NotFound desc = could not find container \"6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087\": container with ID starting with 6dc137082299d6ff834e0cca999a7e45bf0289aea0a27705e13aaa2e86c4e087 not found: ID does not exist" Sep 18 21:38:53.958320 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.958287 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q"] Sep 18 21:38:53.963803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:53.963774 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-5888e6/ipp-route-5888e6-kserve-cf9469b5f-bpz2q"] Sep 18 21:38:55.366613 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:38:55.366578 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24574153-8abb-4119-9441-77b843d3573f" path="/var/lib/kubelet/pods/24574153-8abb-4119-9441-77b843d3573f/volumes" Sep 18 21:39:13.768933 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:13.768900 2577 scope.go:117] "RemoveContainer" containerID="c2ed57e809282fe2715f61d40817b7113d198602e681cc7d040ff5aa7e305ca9" Sep 18 21:39:13.778200 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:13.778175 2577 scope.go:117] "RemoveContainer" containerID="cd7733e38eb05460931aed4c954189e812ca3c6bc80f0383f960ee618234a6a8" Sep 18 21:39:23.073305 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:23.073274 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz"] Sep 18 21:39:23.073879 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:23.073509 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" containerID="cri-o://d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f" gracePeriod=30 Sep 18 21:39:28.335450 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.335379 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:39:28.429506 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429473 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-token\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.429506 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429517 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-data\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.429743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429558 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-socket\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.429743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429581 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-452ms\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-kube-api-access-452ms\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.429743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429663 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-credential-socket\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.429743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429695 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-envoy\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.429743 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429727 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istiod-ca-cert\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.430029 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429785 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-podinfo\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.430029 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429823 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-certs\") pod \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\" (UID: \"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0\") " Sep 18 21:39:28.430029 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429880 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:28.430029 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.429895 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:28.430256 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.430040 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:28.430256 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.430175 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:39:28.430843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.430380 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:28.430843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.430482 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:28.430843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.430499 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:28.430843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.430513 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:28.430843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.430526 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:28.430843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.430539 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:28.432084 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.432054 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-kube-api-access-452ms" (OuterVolumeSpecName: "kube-api-access-452ms") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "kube-api-access-452ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:39:28.432206 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.432133 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:39:28.432272 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.432259 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:39:28.432534 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.432508 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" (UID: "a7a34c13-ead1-4fb1-8552-8aeed5b3cec0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:28.531351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.531306 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-452ms\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-kube-api-access-452ms\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:28.531351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.531342 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:28.531351 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.531354 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:28.531617 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:28.531366 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:29.164220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.164182 2577 generic.go:356] "Generic (PLEG): container finished" podID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerID="d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f" exitCode=0 Sep 18 21:39:29.164401 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.164230 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" Sep 18 21:39:29.164401 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.164266 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" event={"ID":"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0","Type":"ContainerDied","Data":"d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f"} Sep 18 21:39:29.164401 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.164322 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz" event={"ID":"a7a34c13-ead1-4fb1-8552-8aeed5b3cec0","Type":"ContainerDied","Data":"4e8db9181f40fb8cc36d2873b9361f223ab90ac8dbdd7e94a4e8fd8a1b47880b"} Sep 18 21:39:29.164401 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.164353 2577 scope.go:117] "RemoveContainer" containerID="d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f" Sep 18 21:39:29.175155 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.175133 2577 scope.go:117] "RemoveContainer" containerID="8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d" Sep 18 21:39:29.187373 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.187345 2577 scope.go:117] "RemoveContainer" containerID="d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f" Sep 18 21:39:29.187721 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:39:29.187694 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f\": container with ID starting with d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f not found: ID does not exist" containerID="d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f" Sep 18 21:39:29.187776 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.187733 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f"} err="failed to get container status \"d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f\": rpc error: code = NotFound desc = could not find container \"d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f\": container with ID starting with d79e5d5632c3fb4aad5c97e1968feec87eb0491f664aceb526435c8769d4008f not found: ID does not exist" Sep 18 21:39:29.187776 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.187754 2577 scope.go:117] "RemoveContainer" containerID="8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d" Sep 18 21:39:29.188106 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:39:29.188080 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d\": container with ID starting with 8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d not found: ID does not exist" containerID="8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d" Sep 18 21:39:29.188202 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.188109 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d"} err="failed to get container status \"8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d\": rpc error: code = NotFound desc = could not find container \"8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d\": container with ID starting with 8e9fc7adfcd40f05dc66db547d9711fcc7555f080b7d44e52f514acd0adaf49d not found: ID does not exist" Sep 18 21:39:29.190265 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.190243 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz"] Sep 18 21:39:29.197493 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.197467 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5888e6-openshift-default-6747cfdd94-cn7mz"] Sep 18 21:39:29.359172 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:29.359136 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" path="/var/lib/kubelet/pods/a7a34c13-ead1-4fb1-8552-8aeed5b3cec0/volumes" Sep 18 21:39:42.599567 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:42.599532 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z"] Sep 18 21:39:42.600180 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:42.599742 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" containerID="cri-o://7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590" gracePeriod=30 Sep 18 21:39:47.850837 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.850806 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:39:47.928985 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.928896 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bszpf\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-kube-api-access-bszpf\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.928985 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.928937 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-token\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.929213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.928990 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930099c3-dddd-4873-82ee-7f5de6a1a64d-istiod-ca-cert\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.929213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929026 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-data\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.929213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929078 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-socket\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.929213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-podinfo\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.929213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929137 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-certs\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.929213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929154 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-credential-socket\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.929213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929179 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-envoy\") pod \"930099c3-dddd-4873-82ee-7f5de6a1a64d\" (UID: \"930099c3-dddd-4873-82ee-7f5de6a1a64d\") " Sep 18 21:39:47.929581 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929353 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:47.929581 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929427 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930099c3-dddd-4873-82ee-7f5de6a1a64d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:39:47.929581 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929562 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:47.929752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929577 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:47.929752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929637 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-data" (OuterVolumeSpecName: "istio-data") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:47.929752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.929698 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:47.931292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.931266 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-kube-api-access-bszpf" (OuterVolumeSpecName: "kube-api-access-bszpf") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "kube-api-access-bszpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:39:47.931292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.931282 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:39:47.931627 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.931602 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:39:47.931627 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:47.931615 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-token" (OuterVolumeSpecName: "istio-token") pod "930099c3-dddd-4873-82ee-7f5de6a1a64d" (UID: "930099c3-dddd-4873-82ee-7f5de6a1a64d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:39:48.030939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.030906 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:48.030939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.030936 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:48.030939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.030946 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:48.030939 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.030976 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:48.031333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.031008 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bszpf\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-kube-api-access-bszpf\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:48.031333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.031018 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:48.031333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.031027 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/930099c3-dddd-4873-82ee-7f5de6a1a64d-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:48.031333 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.031035 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/930099c3-dddd-4873-82ee-7f5de6a1a64d-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:39:48.281999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.281892 2577 generic.go:356] "Generic (PLEG): container finished" podID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerID="7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590" exitCode=0 Sep 18 21:39:48.281999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.281938 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" Sep 18 21:39:48.281999 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.281979 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" event={"ID":"930099c3-dddd-4873-82ee-7f5de6a1a64d","Type":"ContainerDied","Data":"7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590"} Sep 18 21:39:48.282247 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.282036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z" event={"ID":"930099c3-dddd-4873-82ee-7f5de6a1a64d","Type":"ContainerDied","Data":"9c49477246a842f300cb24861f1ede06a51b69d5545005d01572b992b278e1cb"} Sep 18 21:39:48.282247 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.282083 2577 scope.go:117] "RemoveContainer" containerID="7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590" Sep 18 21:39:48.292338 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.292320 2577 scope.go:117] "RemoveContainer" containerID="8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98" Sep 18 21:39:48.300734 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.300706 2577 scope.go:117] "RemoveContainer" containerID="7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590" Sep 18 21:39:48.301005 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:39:48.300985 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590\": container with ID starting with 7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590 not found: ID does not exist" containerID="7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590" Sep 18 21:39:48.301085 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.301014 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590"} err="failed to get container status \"7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590\": rpc error: code = NotFound desc = could not find container \"7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590\": container with ID starting with 7575b5462f44445f9885dcbcf6e183189902b736fe850dae72a6a197086ea590 not found: ID does not exist" Sep 18 21:39:48.301085 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.301030 2577 scope.go:117] "RemoveContainer" containerID="8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98" Sep 18 21:39:48.301284 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:39:48.301265 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98\": container with ID starting with 8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98 not found: ID does not exist" containerID="8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98" Sep 18 21:39:48.301341 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.301293 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98"} err="failed to get container status \"8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98\": rpc error: code = NotFound desc = could not find container \"8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98\": container with ID starting with 8fc23f8883a8c741591cc996bfad4c98a1c46f27f3f9bfc0063271665890cc98 not found: ID does not exist" Sep 18 21:39:48.312220 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.312193 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z"] Sep 18 21:39:48.317007 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:48.316982 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-deac7a-openshift-default-54c8b9f594-hxz9z"] Sep 18 21:39:49.359804 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:39:49.359771 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" path="/var/lib/kubelet/pods/930099c3-dddd-4873-82ee-7f5de6a1a64d/volumes" Sep 18 21:40:01.570697 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:01.570644 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t"] Sep 18 21:40:01.571253 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:01.570883 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" containerID="cri-o://785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562" gracePeriod=30 Sep 18 21:40:03.628285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:03.628247 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 21:40:03.628673 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:03.628316 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 21:40:06.830657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.830630 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:40:06.931573 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931469 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-data\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931573 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931512 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a84485-4eb8-46de-8bb0-ffd9564a575b-istiod-ca-cert\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931573 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931548 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-token\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931573 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931571 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dt6z\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-kube-api-access-9dt6z\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931593 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-envoy\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931614 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-socket\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931679 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-credential-socket\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931707 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-podinfo\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931734 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-certs\") pod \"81a84485-4eb8-46de-8bb0-ffd9564a575b\" (UID: \"81a84485-4eb8-46de-8bb0-ffd9564a575b\") " Sep 18 21:40:06.931930 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.931804 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-data" (OuterVolumeSpecName: "istio-data") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:06.932323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.932020 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:06.932323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.932061 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:06.932323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.932090 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a84485-4eb8-46de-8bb0-ffd9564a575b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:40:06.932323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.932157 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:06.932323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.932173 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:06.932323 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.932206 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:06.934094 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.934067 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:06.934213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.934157 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:40:06.934213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.934150 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-token" (OuterVolumeSpecName: "istio-token") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:40:06.934213 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:06.934164 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-kube-api-access-9dt6z" (OuterVolumeSpecName: "kube-api-access-9dt6z") pod "81a84485-4eb8-46de-8bb0-ffd9564a575b" (UID: "81a84485-4eb8-46de-8bb0-ffd9564a575b"). InnerVolumeSpecName "kube-api-access-9dt6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:40:07.033329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.033279 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a84485-4eb8-46de-8bb0-ffd9564a575b-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:07.033329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.033325 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:07.033329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.033335 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dt6z\" (UniqueName: \"kubernetes.io/projected/81a84485-4eb8-46de-8bb0-ffd9564a575b-kube-api-access-9dt6z\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:07.033329 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.033345 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:07.033624 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.033354 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:07.033624 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.033362 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a84485-4eb8-46de-8bb0-ffd9564a575b-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:07.033624 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.033371 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a84485-4eb8-46de-8bb0-ffd9564a575b-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:07.402507 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.402479 2577 generic.go:356] "Generic (PLEG): container finished" podID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerID="785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562" exitCode=0 Sep 18 21:40:07.402651 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.402522 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" Sep 18 21:40:07.402651 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.402561 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" event={"ID":"81a84485-4eb8-46de-8bb0-ffd9564a575b","Type":"ContainerDied","Data":"785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562"} Sep 18 21:40:07.402651 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.402613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t" event={"ID":"81a84485-4eb8-46de-8bb0-ffd9564a575b","Type":"ContainerDied","Data":"89dafe0fb9bea00258e7746241171460b89c51c94a530b51aee91b4566513247"} Sep 18 21:40:07.402651 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.402635 2577 scope.go:117] "RemoveContainer" containerID="785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562" Sep 18 21:40:07.411874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.411849 2577 scope.go:117] "RemoveContainer" containerID="ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb" Sep 18 21:40:07.419985 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.419967 2577 scope.go:117] "RemoveContainer" containerID="785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562" Sep 18 21:40:07.420247 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:40:07.420225 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562\": container with ID starting with 785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562 not found: ID does not exist" containerID="785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562" Sep 18 21:40:07.420313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.420254 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562"} err="failed to get container status \"785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562\": rpc error: code = NotFound desc = could not find container \"785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562\": container with ID starting with 785ae58d93372c93335a4fff61b7992dbd3dbc25068202b76e8c0d3b32016562 not found: ID does not exist" Sep 18 21:40:07.420313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.420271 2577 scope.go:117] "RemoveContainer" containerID="ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb" Sep 18 21:40:07.420513 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:40:07.420499 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb\": container with ID starting with ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb not found: ID does not exist" containerID="ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb" Sep 18 21:40:07.420555 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.420517 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb"} err="failed to get container status \"ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb\": rpc error: code = NotFound desc = could not find container \"ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb\": container with ID starting with ef62e090c04839f1473884d7995ba0708f22cf3442d04bd892a9d0982986c4bb not found: ID does not exist" Sep 18 21:40:07.427948 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.427927 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t"] Sep 18 21:40:07.433969 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:07.433935 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cad5cf-openshift-default-54c5d75b88-wm55t"] Sep 18 21:40:09.359775 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:09.359728 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" path="/var/lib/kubelet/pods/81a84485-4eb8-46de-8bb0-ffd9564a575b/volumes" Sep 18 21:40:21.716369 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:21.716331 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl"] Sep 18 21:40:21.716940 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:21.716551 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" containerID="cri-o://14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737" gracePeriod=30 Sep 18 21:40:26.218202 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:26.218112 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 21:40:26.218202 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:26.218178 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 21:40:27.000653 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.000584 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:40:27.012292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012250 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-socket\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012292 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012292 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-credential-socket\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012343 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-certs\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012397 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7trf\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-kube-api-access-m7trf\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012431 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-data\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012467 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istiod-ca-cert\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012527 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012524 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-token\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012792 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012544 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:27.012792 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:27.012792 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012557 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-podinfo\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012792 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012611 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-envoy\") pod \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\" (UID: \"afd3ac50-cc93-4b7d-899a-6c1a23901dfa\") " Sep 18 21:40:27.012792 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012672 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-data" (OuterVolumeSpecName: "istio-data") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:27.012792 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012706 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:27.013124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.012989 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.013124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.013010 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-credential-socket\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.013124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.013023 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-workload-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.013124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.013036 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-data\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.013124 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.013041 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 21:40:27.014899 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.014868 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-kube-api-access-m7trf" (OuterVolumeSpecName: "kube-api-access-m7trf") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "kube-api-access-m7trf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:40:27.015175 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.015156 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:40:27.015269 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.015175 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 21:40:27.015269 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.015229 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-token" (OuterVolumeSpecName: "istio-token") pod "afd3ac50-cc93-4b7d-899a-6c1a23901dfa" (UID: "afd3ac50-cc93-4b7d-899a-6c1a23901dfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:40:27.114166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.114122 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istiod-ca-cert\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.114166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.114159 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-token\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.114166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.114170 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-podinfo\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.114427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.114181 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-istio-envoy\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.114427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.114194 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7trf\" (UniqueName: \"kubernetes.io/projected/afd3ac50-cc93-4b7d-899a-6c1a23901dfa-kube-api-access-m7trf\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:40:27.532181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.532146 2577 generic.go:356] "Generic (PLEG): container finished" podID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerID="14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737" exitCode=0 Sep 18 21:40:27.532181 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.532194 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" Sep 18 21:40:27.532741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.532228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" event={"ID":"afd3ac50-cc93-4b7d-899a-6c1a23901dfa","Type":"ContainerDied","Data":"14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737"} Sep 18 21:40:27.532741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.532278 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl" event={"ID":"afd3ac50-cc93-4b7d-899a-6c1a23901dfa","Type":"ContainerDied","Data":"271c3d3424bef9edef64b67cada30b7763a1541027d0638c9d6c99e8a0339533"} Sep 18 21:40:27.532741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.532299 2577 scope.go:117] "RemoveContainer" containerID="14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737" Sep 18 21:40:27.541910 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.541888 2577 scope.go:117] "RemoveContainer" containerID="bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021" Sep 18 21:40:27.550739 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.550714 2577 scope.go:117] "RemoveContainer" containerID="14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737" Sep 18 21:40:27.551065 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:40:27.551041 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737\": container with ID starting with 14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737 not found: ID does not exist" containerID="14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737" Sep 18 21:40:27.551198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.551075 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737"} err="failed to get container status \"14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737\": rpc error: code = NotFound desc = could not find container \"14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737\": container with ID starting with 14536d3771dab79dfb5bbc02aa2eddbc3c459c35fccee27a7f5f0c575f2fc737 not found: ID does not exist" Sep 18 21:40:27.551198 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.551094 2577 scope.go:117] "RemoveContainer" containerID="bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021" Sep 18 21:40:27.551354 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:40:27.551340 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021\": container with ID starting with bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021 not found: ID does not exist" containerID="bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021" Sep 18 21:40:27.551354 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.551360 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021"} err="failed to get container status \"bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021\": rpc error: code = NotFound desc = could not find container \"bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021\": container with ID starting with bbc01504963650f15c88f0f2b59d266e9f218bf3193b0b3bd737dfc7c6656021 not found: ID does not exist" Sep 18 21:40:27.563186 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.563159 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl"] Sep 18 21:40:27.569471 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:27.569436 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd993e-openshift-default-6c44fd6d78-4w8bl"] Sep 18 21:40:29.358872 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:40:29.358827 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" path="/var/lib/kubelet/pods/afd3ac50-cc93-4b7d-899a-6c1a23901dfa/volumes" Sep 18 21:41:13.543998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:41:13.543950 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:41:13.551584 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:41:13.551562 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:43:18.315020 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.314983 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk"] Sep 18 21:43:18.315583 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.315189 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" podUID="aefa99f6-20fd-4967-a7a6-dd44d9969866" containerName="manager" containerID="cri-o://8beaaf1dbb0dc2cd386652b4a181b3478153ac67d3e0ebde506e1b228912d820" gracePeriod=10 Sep 18 21:43:18.557018 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.556986 2577 generic.go:356] "Generic (PLEG): container finished" podID="aefa99f6-20fd-4967-a7a6-dd44d9969866" containerID="8beaaf1dbb0dc2cd386652b4a181b3478153ac67d3e0ebde506e1b228912d820" exitCode=0 Sep 18 21:43:18.557169 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.557088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" event={"ID":"aefa99f6-20fd-4967-a7a6-dd44d9969866","Type":"ContainerDied","Data":"8beaaf1dbb0dc2cd386652b4a181b3478153ac67d3e0ebde506e1b228912d820"} Sep 18 21:43:18.573363 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.573303 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:43:18.642319 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.642287 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjnmj\" (UniqueName: \"kubernetes.io/projected/aefa99f6-20fd-4967-a7a6-dd44d9969866-kube-api-access-pjnmj\") pod \"aefa99f6-20fd-4967-a7a6-dd44d9969866\" (UID: \"aefa99f6-20fd-4967-a7a6-dd44d9969866\") " Sep 18 21:43:18.642506 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.642403 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aefa99f6-20fd-4967-a7a6-dd44d9969866-extensions-socket-volume\") pod \"aefa99f6-20fd-4967-a7a6-dd44d9969866\" (UID: \"aefa99f6-20fd-4967-a7a6-dd44d9969866\") " Sep 18 21:43:18.642815 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.642789 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefa99f6-20fd-4967-a7a6-dd44d9969866-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aefa99f6-20fd-4967-a7a6-dd44d9969866" (UID: "aefa99f6-20fd-4967-a7a6-dd44d9969866"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:43:18.644370 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.644337 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefa99f6-20fd-4967-a7a6-dd44d9969866-kube-api-access-pjnmj" (OuterVolumeSpecName: "kube-api-access-pjnmj") pod "aefa99f6-20fd-4967-a7a6-dd44d9969866" (UID: "aefa99f6-20fd-4967-a7a6-dd44d9969866"). InnerVolumeSpecName "kube-api-access-pjnmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:43:18.743841 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.743812 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjnmj\" (UniqueName: \"kubernetes.io/projected/aefa99f6-20fd-4967-a7a6-dd44d9969866-kube-api-access-pjnmj\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:43:18.743841 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:18.743840 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aefa99f6-20fd-4967-a7a6-dd44d9969866-extensions-socket-volume\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:43:19.565492 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:19.565459 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" Sep 18 21:43:19.565492 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:19.565486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk" event={"ID":"aefa99f6-20fd-4967-a7a6-dd44d9969866","Type":"ContainerDied","Data":"63a61385964d27cf3d2b0f5ca28b0985ab9ea0afaa256c7f0324796416738a28"} Sep 18 21:43:19.566035 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:19.565542 2577 scope.go:117] "RemoveContainer" containerID="8beaaf1dbb0dc2cd386652b4a181b3478153ac67d3e0ebde506e1b228912d820" Sep 18 21:43:19.592157 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:19.592125 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk"] Sep 18 21:43:19.599902 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:19.599876 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jrzk"] Sep 18 21:43:21.358633 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:43:21.358594 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aefa99f6-20fd-4967-a7a6-dd44d9969866" path="/var/lib/kubelet/pods/aefa99f6-20fd-4967-a7a6-dd44d9969866/volumes" Sep 18 21:44:13.868712 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:13.868670 2577 scope.go:117] "RemoveContainer" containerID="ddfca5dad4e641b2272fcdb57de9c9d775fbe1003bb7f60b964e72699c3729bc" Sep 18 21:44:19.010387 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010348 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq"] Sep 18 21:44:19.010825 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010810 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" Sep 18 21:44:19.010872 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010829 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" Sep 18 21:44:19.010872 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010848 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" Sep 18 21:44:19.010872 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010855 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" Sep 18 21:44:19.010872 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010871 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010878 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010890 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefa99f6-20fd-4967-a7a6-dd44d9969866" containerName="manager" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010898 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefa99f6-20fd-4967-a7a6-dd44d9969866" containerName="manager" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010906 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010913 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010922 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010928 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010935 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010941 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010964 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010972 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010985 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24574153-8abb-4119-9441-77b843d3573f" containerName="main" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.010990 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24574153-8abb-4119-9441-77b843d3573f" containerName="main" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011001 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011009 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011019 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef6cb65-60c4-4eab-9692-0a1db95ed057" containerName="revoke-keys" Sep 18 21:44:19.011039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011027 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef6cb65-60c4-4eab-9692-0a1db95ed057" containerName="revoke-keys" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011118 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011129 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011139 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefa99f6-20fd-4967-a7a6-dd44d9969866" containerName="manager" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011146 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011153 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011161 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011168 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd3ac50-cc93-4b7d-899a-6c1a23901dfa" containerName="istio-proxy" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011175 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24574153-8abb-4119-9441-77b843d3573f" containerName="main" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011184 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5150ffbc-b2f7-4a65-94e2-beea89b65129" containerName="istio-proxy" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011193 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011201 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef6cb65-60c4-4eab-9692-0a1db95ed057" containerName="revoke-keys" Sep 18 21:44:19.011553 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.011210 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a34c13-ead1-4fb1-8552-8aeed5b3cec0" containerName="istio-proxy" Sep 18 21:44:19.015781 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.015761 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.018467 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.018445 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zlvvm\"" Sep 18 21:44:19.028325 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.028301 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq"] Sep 18 21:44:19.134865 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.134824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8857a6ab-cbda-4ae3-bc67-2047c696bfc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c2xnq\" (UID: \"8857a6ab-cbda-4ae3-bc67-2047c696bfc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.135075 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.134946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhnf\" (UniqueName: \"kubernetes.io/projected/8857a6ab-cbda-4ae3-bc67-2047c696bfc1-kube-api-access-bjhnf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c2xnq\" (UID: \"8857a6ab-cbda-4ae3-bc67-2047c696bfc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.236176 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.236136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhnf\" (UniqueName: \"kubernetes.io/projected/8857a6ab-cbda-4ae3-bc67-2047c696bfc1-kube-api-access-bjhnf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c2xnq\" (UID: \"8857a6ab-cbda-4ae3-bc67-2047c696bfc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.236356 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.236214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8857a6ab-cbda-4ae3-bc67-2047c696bfc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c2xnq\" (UID: \"8857a6ab-cbda-4ae3-bc67-2047c696bfc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.236610 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.236589 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8857a6ab-cbda-4ae3-bc67-2047c696bfc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c2xnq\" (UID: \"8857a6ab-cbda-4ae3-bc67-2047c696bfc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.245103 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.245079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhnf\" (UniqueName: \"kubernetes.io/projected/8857a6ab-cbda-4ae3-bc67-2047c696bfc1-kube-api-access-bjhnf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-c2xnq\" (UID: \"8857a6ab-cbda-4ae3-bc67-2047c696bfc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.326250 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.326220 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.615020 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.614990 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq"] Sep 18 21:44:19.616770 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:44:19.616745 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8857a6ab_cbda_4ae3_bc67_2047c696bfc1.slice/crio-b095da65609f2133b6740e98e6bff5ce04e9edbcc204d4a417387b1d1e4836d5 WatchSource:0}: Error finding container b095da65609f2133b6740e98e6bff5ce04e9edbcc204d4a417387b1d1e4836d5: Status 404 returned error can't find the container with id b095da65609f2133b6740e98e6bff5ce04e9edbcc204d4a417387b1d1e4836d5 Sep 18 21:44:19.917678 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.917587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" event={"ID":"8857a6ab-cbda-4ae3-bc67-2047c696bfc1","Type":"ContainerStarted","Data":"0efc6d40cbaa1693ada8c7372a845fade112cf07b505b6a1ecf4eb9c28186216"} Sep 18 21:44:19.917678 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.917644 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" event={"ID":"8857a6ab-cbda-4ae3-bc67-2047c696bfc1","Type":"ContainerStarted","Data":"b095da65609f2133b6740e98e6bff5ce04e9edbcc204d4a417387b1d1e4836d5"} Sep 18 21:44:19.917866 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.917679 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:44:19.943143 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:19.943092 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" podStartSLOduration=1.943073122 podStartE2EDuration="1.943073122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:44:19.940241259 +0000 UTC m=+2288.869987133" watchObservedRunningTime="2026-09-18 21:44:19.943073122 +0000 UTC m=+2288.872818995" Sep 18 21:44:30.925053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:44:30.925021 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-c2xnq" Sep 18 21:45:09.944469 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.944433 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2"] Sep 18 21:45:09.944918 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.944825 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" Sep 18 21:45:09.944918 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.944836 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="930099c3-dddd-4873-82ee-7f5de6a1a64d" containerName="istio-proxy" Sep 18 21:45:09.944918 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.944862 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" Sep 18 21:45:09.944918 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.944868 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" Sep 18 21:45:09.945083 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.944924 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a84485-4eb8-46de-8bb0-ffd9564a575b" containerName="istio-proxy" Sep 18 21:45:09.947868 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.947853 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:09.950588 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.950562 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-14f9af6a-kserve-self-signed-certs\"" Sep 18 21:45:09.978645 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:09.978616 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2"] Sep 18 21:45:10.096411 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.096372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-home\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.096587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.096438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gj8p\" (UniqueName: \"kubernetes.io/projected/54e5587b-99ce-493b-8e1e-6d8c316c3683-kube-api-access-9gj8p\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.096587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.096477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-model-cache\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.096587 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.096554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e5587b-99ce-493b-8e1e-6d8c316c3683-tls-certs\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.096715 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.096642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-tmp-dir\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.096715 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.096693 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-dshm\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.197945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.197856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj8p\" (UniqueName: \"kubernetes.io/projected/54e5587b-99ce-493b-8e1e-6d8c316c3683-kube-api-access-9gj8p\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.197945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.197901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-model-cache\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.197945 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.197932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e5587b-99ce-493b-8e1e-6d8c316c3683-tls-certs\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.198285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.198005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-tmp-dir\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.198285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.198044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-dshm\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.198285 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.198103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-home\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.198468 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.198413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-model-cache\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.198468 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.198447 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-tmp-dir\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.198545 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.198473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-home\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.200423 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.200401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-dshm\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.200803 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.200782 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e5587b-99ce-493b-8e1e-6d8c316c3683-tls-certs\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.210286 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.210264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gj8p\" (UniqueName: \"kubernetes.io/projected/54e5587b-99ce-493b-8e1e-6d8c316c3683-kube-api-access-9gj8p\") pod \"e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.257423 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.257351 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:10.541716 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:10.541693 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2"] Sep 18 21:45:10.543853 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:45:10.543825 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54e5587b_99ce_493b_8e1e_6d8c316c3683.slice/crio-50361f34bfe4119b0abcbaaff409248623930d6fc3302a53675ad645b55d535b WatchSource:0}: Error finding container 50361f34bfe4119b0abcbaaff409248623930d6fc3302a53675ad645b55d535b: Status 404 returned error can't find the container with id 50361f34bfe4119b0abcbaaff409248623930d6fc3302a53675ad645b55d535b Sep 18 21:45:11.235760 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:11.235715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" event={"ID":"54e5587b-99ce-493b-8e1e-6d8c316c3683","Type":"ContainerStarted","Data":"e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e"} Sep 18 21:45:11.236219 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:11.235772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" event={"ID":"54e5587b-99ce-493b-8e1e-6d8c316c3683","Type":"ContainerStarted","Data":"50361f34bfe4119b0abcbaaff409248623930d6fc3302a53675ad645b55d535b"} Sep 18 21:45:11.304151 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:11.304082 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" podStartSLOduration=2.304059854 podStartE2EDuration="2.304059854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:45:11.301942805 +0000 UTC m=+2340.231688685" watchObservedRunningTime="2026-09-18 21:45:11.304059854 +0000 UTC m=+2340.233805727" Sep 18 21:45:20.258150 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:20.258111 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:20.258561 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:20.258163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:20.271024 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:20.270999 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:20.300188 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:20.300160 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:29.918394 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:29.918358 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2"] Sep 18 21:45:44.944127 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:44.944081 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" podUID="54e5587b-99ce-493b-8e1e-6d8c316c3683" containerName="main" containerID="cri-o://e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e" gracePeriod=45 Sep 18 21:45:45.194837 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.194780 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:45.327936 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.327888 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-tmp-dir\") pod \"54e5587b-99ce-493b-8e1e-6d8c316c3683\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " Sep 18 21:45:45.327936 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.327928 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-dshm\") pod \"54e5587b-99ce-493b-8e1e-6d8c316c3683\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " Sep 18 21:45:45.328227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-model-cache\") pod \"54e5587b-99ce-493b-8e1e-6d8c316c3683\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " Sep 18 21:45:45.328227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328035 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e5587b-99ce-493b-8e1e-6d8c316c3683-tls-certs\") pod \"54e5587b-99ce-493b-8e1e-6d8c316c3683\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " Sep 18 21:45:45.328227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328092 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-home\") pod \"54e5587b-99ce-493b-8e1e-6d8c316c3683\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " Sep 18 21:45:45.328227 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328121 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gj8p\" (UniqueName: \"kubernetes.io/projected/54e5587b-99ce-493b-8e1e-6d8c316c3683-kube-api-access-9gj8p\") pod \"54e5587b-99ce-493b-8e1e-6d8c316c3683\" (UID: \"54e5587b-99ce-493b-8e1e-6d8c316c3683\") " Sep 18 21:45:45.328437 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328239 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54e5587b-99ce-493b-8e1e-6d8c316c3683" (UID: "54e5587b-99ce-493b-8e1e-6d8c316c3683"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:45:45.328437 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328399 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-home" (OuterVolumeSpecName: "home") pod "54e5587b-99ce-493b-8e1e-6d8c316c3683" (UID: "54e5587b-99ce-493b-8e1e-6d8c316c3683"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:45:45.328542 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328492 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-tmp-dir\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:45:45.328542 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328511 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-home\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:45:45.328645 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.328617 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-model-cache" (OuterVolumeSpecName: "model-cache") pod "54e5587b-99ce-493b-8e1e-6d8c316c3683" (UID: "54e5587b-99ce-493b-8e1e-6d8c316c3683"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:45:45.330681 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.330641 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e5587b-99ce-493b-8e1e-6d8c316c3683-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54e5587b-99ce-493b-8e1e-6d8c316c3683" (UID: "54e5587b-99ce-493b-8e1e-6d8c316c3683"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 21:45:45.330908 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.330883 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-dshm" (OuterVolumeSpecName: "dshm") pod "54e5587b-99ce-493b-8e1e-6d8c316c3683" (UID: "54e5587b-99ce-493b-8e1e-6d8c316c3683"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 21:45:45.331005 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.330914 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e5587b-99ce-493b-8e1e-6d8c316c3683-kube-api-access-9gj8p" (OuterVolumeSpecName: "kube-api-access-9gj8p") pod "54e5587b-99ce-493b-8e1e-6d8c316c3683" (UID: "54e5587b-99ce-493b-8e1e-6d8c316c3683"). InnerVolumeSpecName "kube-api-access-9gj8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 21:45:45.429616 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.429354 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gj8p\" (UniqueName: \"kubernetes.io/projected/54e5587b-99ce-493b-8e1e-6d8c316c3683-kube-api-access-9gj8p\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:45:45.429616 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.429385 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-dshm\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:45:45.429616 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.429398 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e5587b-99ce-493b-8e1e-6d8c316c3683-model-cache\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:45:45.429616 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.429412 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e5587b-99ce-493b-8e1e-6d8c316c3683-tls-certs\") on node \"ip-10-0-132-187.ec2.internal\" DevicePath \"\"" Sep 18 21:45:45.430807 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.430773 2577 generic.go:356] "Generic (PLEG): container finished" podID="54e5587b-99ce-493b-8e1e-6d8c316c3683" containerID="e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e" exitCode=0 Sep 18 21:45:45.430927 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.430827 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" Sep 18 21:45:45.430927 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.430887 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" event={"ID":"54e5587b-99ce-493b-8e1e-6d8c316c3683","Type":"ContainerDied","Data":"e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e"} Sep 18 21:45:45.431045 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.430941 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2" event={"ID":"54e5587b-99ce-493b-8e1e-6d8c316c3683","Type":"ContainerDied","Data":"50361f34bfe4119b0abcbaaff409248623930d6fc3302a53675ad645b55d535b"} Sep 18 21:45:45.431045 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.430974 2577 scope.go:117] "RemoveContainer" containerID="e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e" Sep 18 21:45:45.441662 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.441638 2577 scope.go:117] "RemoveContainer" containerID="e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e" Sep 18 21:45:45.441987 ip-10-0-132-187 kubenswrapper[2577]: E0918 21:45:45.441938 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e\": container with ID starting with e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e not found: ID does not exist" containerID="e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e" Sep 18 21:45:45.442070 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.441982 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e"} err="failed to get container status \"e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e\": rpc error: code = NotFound desc = could not find container \"e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e\": container with ID starting with e766b5c71155940347fb0deb028da6157b6d0b88b5643c7f6827ac6846ea7d6e not found: ID does not exist" Sep 18 21:45:45.463696 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.463610 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2"] Sep 18 21:45:45.484241 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:45.484205 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-14f9af6a-kserve-584d9b9d8c-w5lb2"] Sep 18 21:45:47.359390 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:47.359358 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54e5587b-99ce-493b-8e1e-6d8c316c3683" path="/var/lib/kubelet/pods/54e5587b-99ce-493b-8e1e-6d8c316c3683/volumes" Sep 18 21:45:53.543057 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.543020 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w"] Sep 18 21:45:53.543465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.543443 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e5587b-99ce-493b-8e1e-6d8c316c3683" containerName="main" Sep 18 21:45:53.543465 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.543454 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e5587b-99ce-493b-8e1e-6d8c316c3683" containerName="main" Sep 18 21:45:53.543546 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.543524 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="54e5587b-99ce-493b-8e1e-6d8c316c3683" containerName="main" Sep 18 21:45:53.548138 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.548121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.551801 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.551779 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 21:45:53.552599 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.552582 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 18 21:45:53.554229 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.554211 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dcnql\"" Sep 18 21:45:53.554325 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.554293 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 18 21:45:53.555149 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.554937 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 18 21:45:53.628189 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.628159 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w"] Sep 18 21:45:53.714474 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.714429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrh8\" (UniqueName: \"kubernetes.io/projected/3a26006b-0c7d-4795-93dd-68366dec0268-kube-api-access-vjrh8\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.714657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.714500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a26006b-0c7d-4795-93dd-68366dec0268-metrics-certs\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.714657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.714550 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a26006b-0c7d-4795-93dd-68366dec0268-openshift-service-ca\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.714657 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.714631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a26006b-0c7d-4795-93dd-68366dec0268-maas-api-tls\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.816032 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.815994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a26006b-0c7d-4795-93dd-68366dec0268-maas-api-tls\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.816233 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.816075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrh8\" (UniqueName: \"kubernetes.io/projected/3a26006b-0c7d-4795-93dd-68366dec0268-kube-api-access-vjrh8\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.816233 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.816109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a26006b-0c7d-4795-93dd-68366dec0268-metrics-certs\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.816233 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.816142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a26006b-0c7d-4795-93dd-68366dec0268-openshift-service-ca\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.816737 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.816710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a26006b-0c7d-4795-93dd-68366dec0268-openshift-service-ca\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.818658 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.818631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a26006b-0c7d-4795-93dd-68366dec0268-maas-api-tls\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.818658 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.818654 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a26006b-0c7d-4795-93dd-68366dec0268-metrics-certs\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.829071 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.829046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrh8\" (UniqueName: \"kubernetes.io/projected/3a26006b-0c7d-4795-93dd-68366dec0268-kube-api-access-vjrh8\") pod \"maas-api-68c86dfd56-rgc8w\" (UID: \"3a26006b-0c7d-4795-93dd-68366dec0268\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:53.858714 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:53.858677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:54.147006 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:54.145873 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w"] Sep 18 21:45:54.151172 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:54.150208 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 21:45:54.486244 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:54.486151 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" event={"ID":"3a26006b-0c7d-4795-93dd-68366dec0268","Type":"ContainerStarted","Data":"f81fbbe7e56a7e5d9362c84e181e4abf757bcf9ca0f50bce1a857212e94240a0"} Sep 18 21:45:57.510663 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:57.510624 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" event={"ID":"3a26006b-0c7d-4795-93dd-68366dec0268","Type":"ContainerStarted","Data":"196d22d68939bc25e53ed9ac7563e2a91ec185b2b22e064681eb9a10daa48963"} Sep 18 21:45:57.511166 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:57.510686 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:45:57.542163 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:45:57.542115 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" podStartSLOduration=2.172046766 podStartE2EDuration="4.542097613s" totalImagesPullingTime="2.370050847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:45:57.539893326 +0000 UTC m=+2386.469639199" watchObservedRunningTime="2026-09-18 21:45:57.542097613 +0000 UTC m=+2386.471843511" Sep 18 21:46:03.523118 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:03.523086 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-rgc8w" Sep 18 21:46:13.585376 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:13.585347 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:46:13.593641 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:13.593610 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzszz_86bef309-7448-4e16-a947-c916df0fb672/ovn-acl-logging/0.log" Sep 18 21:46:19.415921 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:19.415890 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ffdcd95db-jqtqz_1def2918-6cf0-4249-9895-682e1e78ad78/authorino/0.log" Sep 18 21:46:25.216947 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:25.216860 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d54fj_fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d/manager/0.log" Sep 18 21:46:25.345631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:25.345593 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-snz74_a424b6e3-ada2-49a5-9191-457d24496f54/manager/0.log" Sep 18 21:46:26.225791 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:26.225761 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75f94bc69c-sgfsn_c438df90-6953-4453-8877-bbf1117bd8f7/copy-manifests/0.log" Sep 18 21:46:26.237692 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:26.237667 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75f94bc69c-sgfsn_c438df90-6953-4453-8877-bbf1117bd8f7/manager/0.log" Sep 18 21:46:26.742630 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:26.742597 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68c86dfd56-rgc8w_3a26006b-0c7d-4795-93dd-68366dec0268/maas-api/0.log" Sep 18 21:46:27.931358 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:27.931331 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ffdcd95db-jqtqz_1def2918-6cf0-4249-9895-682e1e78ad78/authorino/0.log" Sep 18 21:46:28.081488 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:28.081452 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dph4s_2c649398-ea3a-4509-b35d-99e6808e00b0/manager/0.log" Sep 18 21:46:28.217077 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:28.217000 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-t26kw_5a2f719b-b21e-45a0-9717-b14ef8628bdb/manager/0.log" Sep 18 21:46:28.340173 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:28.340146 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-g956f_2e6e665b-9f29-4cfc-8dde-80c9639d2fda/kuadrant-console-plugin/0.log" Sep 18 21:46:28.477736 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:28.477655 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-c2xnq_8857a6ab-cbda-4ae3-bc67-2047c696bfc1/manager/0.log" Sep 18 21:46:28.753505 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:28.753428 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-gc4g9_9bc9fac1-f157-4505-85de-16b78d21a1fe/manager/0.log" Sep 18 21:46:29.121717 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:29.121690 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g_26452a17-3521-44a0-88fa-9f1a9cd30684/istio-proxy/0.log" Sep 18 21:46:29.400313 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:29.400239 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85cdd48bb5-v6c8k_fded3273-f36c-45d1-b000-03d913f26bd6/kube-auth-proxy/0.log" Sep 18 21:46:29.646146 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:29.646118 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j9d9x_0da2b319-54e0-4adc-928c-cf7faf5849be/istio-proxy/1.log" Sep 18 21:46:29.785470 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:29.785391 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-nczzc_b0f9126a-2f1a-499c-9e9b-6c5e1854b31a/payload-pre-processing/0.log" Sep 18 21:46:30.584254 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:30.584215 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-95vwf_e887449e-0489-42c4-860a-216547d2a932/main/0.log" Sep 18 21:46:30.727053 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:30.727022 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-v88ls_197d4cbc-63e5-4583-9bfb-1349a0448366/main/0.log" Sep 18 21:46:30.843943 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:30.843866 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-2skvj_60cb8262-6bab-4891-b050-05fc68eac5e7/main/0.log" Sep 18 21:46:39.708881 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:39.708841 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6xgxk_66d8cb46-3ba4-4b18-b769-a7667c20495c/global-pull-secret-syncer/0.log" Sep 18 21:46:39.986670 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:39.986584 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-55w59_69f29fa6-0d0c-487a-ba87-8b99fbf8d364/konnectivity-agent/0.log" Sep 18 21:46:40.164831 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:40.164789 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-187.ec2.internal_8a2162d0f83354e882cf0ecc5729b614/haproxy/0.log" Sep 18 21:46:44.417204 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:44.417172 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-ffdcd95db-jqtqz_1def2918-6cf0-4249-9895-682e1e78ad78/authorino/0.log" Sep 18 21:46:44.517778 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:44.517750 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dph4s_2c649398-ea3a-4509-b35d-99e6808e00b0/manager/0.log" Sep 18 21:46:44.569912 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:44.569886 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-t26kw_5a2f719b-b21e-45a0-9717-b14ef8628bdb/manager/0.log" Sep 18 21:46:44.616461 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:44.616432 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-g956f_2e6e665b-9f29-4cfc-8dde-80c9639d2fda/kuadrant-console-plugin/0.log" Sep 18 21:46:44.690523 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:44.690431 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-c2xnq_8857a6ab-cbda-4ae3-bc67-2047c696bfc1/manager/0.log" Sep 18 21:46:44.864105 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:44.864073 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-gc4g9_9bc9fac1-f157-4505-85de-16b78d21a1fe/manager/0.log" Sep 18 21:46:47.508155 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:47.508123 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-kls8k_b4311bed-80a0-4146-9ec8-ecd4daedaf0c/cluster-monitoring-operator/0.log" Sep 18 21:46:48.135406 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:48.135377 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-hl85b_a8e27e6c-4674-4a31-9898-340876416861/monitoring-plugin/0.log" Sep 18 21:46:48.218582 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:48.218546 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2hwfk_842cb176-3c0b-4588-8409-03422d790d40/node-exporter/0.log" Sep 18 21:46:48.328869 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:48.328831 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2hwfk_842cb176-3c0b-4588-8409-03422d790d40/kube-rbac-proxy/0.log" Sep 18 21:46:48.394141 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:48.394067 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2hwfk_842cb176-3c0b-4588-8409-03422d790d40/init-textfile/0.log" Sep 18 21:46:48.935117 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:48.935090 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-n66rq_260c0408-1d9c-45c8-93ef-54f75c774bfc/kube-rbac-proxy-main/0.log" Sep 18 21:46:48.992349 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:48.992316 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-n66rq_260c0408-1d9c-45c8-93ef-54f75c774bfc/kube-rbac-proxy-self/0.log" Sep 18 21:46:49.046099 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.046063 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6"] Sep 18 21:46:49.049756 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.049738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.054113 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.054090 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s7tjl\"/\"default-dockercfg-jr7nq\"" Sep 18 21:46:49.054813 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.054795 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s7tjl\"/\"kube-root-ca.crt\"" Sep 18 21:46:49.054870 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.054852 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s7tjl\"/\"openshift-service-ca.crt\"" Sep 18 21:46:49.070325 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.070298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6"] Sep 18 21:46:49.102881 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.102829 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-n66rq_260c0408-1d9c-45c8-93ef-54f75c774bfc/openshift-state-metrics/0.log" Sep 18 21:46:49.142752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.142720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-proc\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.142752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.142758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-podres\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.142994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.142789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-sys\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.142994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.142833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-lib-modules\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.142994 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.142941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4tw\" (UniqueName: \"kubernetes.io/projected/f6a965ee-e7d2-45fb-bc99-1b148a02268f-kube-api-access-2g4tw\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.243742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-proc\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.243742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-podres\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.243742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-sys\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.243742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-lib-modules\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.244082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4tw\" (UniqueName: \"kubernetes.io/projected/f6a965ee-e7d2-45fb-bc99-1b148a02268f-kube-api-access-2g4tw\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.244082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243802 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-proc\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.244082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-sys\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.244082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243847 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-podres\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.244082 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.243890 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6a965ee-e7d2-45fb-bc99-1b148a02268f-lib-modules\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.258427 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.258397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4tw\" (UniqueName: \"kubernetes.io/projected/f6a965ee-e7d2-45fb-bc99-1b148a02268f-kube-api-access-2g4tw\") pod \"perf-node-gather-daemonset-w6lx6\" (UID: \"f6a965ee-e7d2-45fb-bc99-1b148a02268f\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.359822 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.359795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:49.700742 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.700708 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6"] Sep 18 21:46:49.701686 ip-10-0-132-187 kubenswrapper[2577]: W0918 21:46:49.701651 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf6a965ee_e7d2_45fb_bc99_1b148a02268f.slice/crio-5c1643e48ea4d1ab253faadddea54e06ee063acf3ca1d7ff68e645fb833db68c WatchSource:0}: Error finding container 5c1643e48ea4d1ab253faadddea54e06ee063acf3ca1d7ff68e645fb833db68c: Status 404 returned error can't find the container with id 5c1643e48ea4d1ab253faadddea54e06ee063acf3ca1d7ff68e645fb833db68c Sep 18 21:46:49.824843 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.824807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" event={"ID":"f6a965ee-e7d2-45fb-bc99-1b148a02268f","Type":"ContainerStarted","Data":"5c1643e48ea4d1ab253faadddea54e06ee063acf3ca1d7ff68e645fb833db68c"} Sep 18 21:46:49.982765 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:49.982674 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-sqn2w_bb7a7257-28fa-483c-9cce-0a00bc5c0f66/prometheus-operator/0.log" Sep 18 21:46:50.070255 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.070228 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-sqn2w_bb7a7257-28fa-483c-9cce-0a00bc5c0f66/kube-rbac-proxy/0.log" Sep 18 21:46:50.475649 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.475609 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97fff58fc-lmwhq_ffc465a4-1baf-41fc-8d98-7f81323c3dc6/thanos-query/0.log" Sep 18 21:46:50.558631 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.558604 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97fff58fc-lmwhq_ffc465a4-1baf-41fc-8d98-7f81323c3dc6/kube-rbac-proxy-web/0.log" Sep 18 21:46:50.601039 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.600937 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97fff58fc-lmwhq_ffc465a4-1baf-41fc-8d98-7f81323c3dc6/kube-rbac-proxy/0.log" Sep 18 21:46:50.647415 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.647386 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97fff58fc-lmwhq_ffc465a4-1baf-41fc-8d98-7f81323c3dc6/prom-label-proxy/0.log" Sep 18 21:46:50.700769 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.700738 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97fff58fc-lmwhq_ffc465a4-1baf-41fc-8d98-7f81323c3dc6/kube-rbac-proxy-rules/0.log" Sep 18 21:46:50.749874 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.749847 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97fff58fc-lmwhq_ffc465a4-1baf-41fc-8d98-7f81323c3dc6/kube-rbac-proxy-metrics/0.log" Sep 18 21:46:50.833787 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.833753 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" event={"ID":"f6a965ee-e7d2-45fb-bc99-1b148a02268f","Type":"ContainerStarted","Data":"45912d2cc29d0cafb166fc12bd6388ca1c0854db5e7614e388432e7351945c67"} Sep 18 21:46:50.833998 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.833866 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:50.874615 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:50.874515 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" podStartSLOduration=1.874499635 podStartE2EDuration="1.874499635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:46:50.872048925 +0000 UTC m=+2439.801794799" watchObservedRunningTime="2026-09-18 21:46:50.874499635 +0000 UTC m=+2439.804245515" Sep 18 21:46:51.940671 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:51.940643 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-tc6hg_6795d358-7432-4dbc-a986-3bb8d282823b/networking-console-plugin/0.log" Sep 18 21:46:53.450115 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:53.450083 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7ddcb54c4b-8qgd4_2fb4badf-4634-4af6-b0ab-a55f38647996/console/0.log" Sep 18 21:46:55.891902 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:55.891873 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hpqpv_b085f059-d211-47ed-9747-bb25ea39e37d/dns/0.log" Sep 18 21:46:55.965280 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:55.965244 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hpqpv_b085f059-d211-47ed-9747-bb25ea39e37d/kube-rbac-proxy/0.log" Sep 18 21:46:56.275228 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:56.275138 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4588v_79faf647-1593-440d-b79d-383738434d06/dns-node-resolver/0.log" Sep 18 21:46:56.850043 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:56.850017 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-w6lx6" Sep 18 21:46:57.089116 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:57.089084 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-867666fdd5-bbs7d_423706e6-0050-48a4-afa1-5263ef5cb93b/registry/0.log" Sep 18 21:46:57.219950 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:57.219868 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9zlx2_23c22257-3ecd-4264-89ef-560ecd2a6c41/node-ca/0.log" Sep 18 21:46:58.967563 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:58.967520 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g_26452a17-3521-44a0-88fa-9f1a9cd30684/istio-proxy/0.log" Sep 18 21:46:59.844577 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:59.844545 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85cdd48bb5-v6c8k_fded3273-f36c-45d1-b000-03d913f26bd6/kube-auth-proxy/0.log" Sep 18 21:46:59.966453 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:59.966421 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j9d9x_0da2b319-54e0-4adc-928c-cf7faf5849be/istio-proxy/0.log" Sep 18 21:46:59.971304 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:46:59.971270 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j9d9x_0da2b319-54e0-4adc-928c-cf7faf5849be/istio-proxy/1.log" Sep 18 21:47:00.043706 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:00.043671 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-nczzc_b0f9126a-2f1a-499c-9e9b-6c5e1854b31a/payload-pre-processing/0.log" Sep 18 21:47:00.476574 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:00.476542 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g_26452a17-3521-44a0-88fa-9f1a9cd30684/istio-proxy/0.log" Sep 18 21:47:01.224026 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:01.223928 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85cdd48bb5-v6c8k_fded3273-f36c-45d1-b000-03d913f26bd6/kube-auth-proxy/0.log" Sep 18 21:47:01.342776 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:01.342738 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j9d9x_0da2b319-54e0-4adc-928c-cf7faf5849be/istio-proxy/0.log" Sep 18 21:47:01.347589 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:01.347542 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j9d9x_0da2b319-54e0-4adc-928c-cf7faf5849be/istio-proxy/1.log" Sep 18 21:47:01.397431 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:01.397402 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-nczzc_b0f9126a-2f1a-499c-9e9b-6c5e1854b31a/payload-pre-processing/0.log" Sep 18 21:47:01.550884 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:01.550806 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dvqc2_f1aca64a-306c-4902-94c5-0f3845ac290b/serve-healthcheck-canary/0.log" Sep 18 21:47:02.319399 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:02.319360 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-m8g7n_2ed5b13a-605a-46fe-a982-2d16520f93ea/insights-operator/0.log" Sep 18 21:47:02.320179 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:02.320150 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-m8g7n_2ed5b13a-605a-46fe-a982-2d16520f93ea/insights-operator/1.log" Sep 18 21:47:02.379310 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:02.379280 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cqmw5_f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec/kube-rbac-proxy/0.log" Sep 18 21:47:02.426741 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:02.426715 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cqmw5_f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec/exporter/0.log" Sep 18 21:47:02.485752 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:02.485726 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cqmw5_f87ef4b1-04e8-4aea-9190-e5a4d0fd74ec/extractor/0.log" Sep 18 21:47:03.074779 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:03.074752 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qqc2g_26452a17-3521-44a0-88fa-9f1a9cd30684/istio-proxy/0.log" Sep 18 21:47:03.629622 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:03.629591 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85cdd48bb5-v6c8k_fded3273-f36c-45d1-b000-03d913f26bd6/kube-auth-proxy/0.log" Sep 18 21:47:03.751744 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:03.751708 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j9d9x_0da2b319-54e0-4adc-928c-cf7faf5849be/istio-proxy/0.log" Sep 18 21:47:03.756489 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:03.756461 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j9d9x_0da2b319-54e0-4adc-928c-cf7faf5849be/istio-proxy/1.log" Sep 18 21:47:03.813047 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:03.813010 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-nczzc_b0f9126a-2f1a-499c-9e9b-6c5e1854b31a/payload-pre-processing/0.log" Sep 18 21:47:05.649235 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:05.649200 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d54fj_fc2eaf8b-1d10-4e13-82ca-6743faa0eb9d/manager/0.log" Sep 18 21:47:06.041520 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:06.041440 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-snz74_a424b6e3-ada2-49a5-9191-457d24496f54/manager/0.log" Sep 18 21:47:06.845888 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:06.845861 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75f94bc69c-sgfsn_c438df90-6953-4453-8877-bbf1117bd8f7/manager/0.log" Sep 18 21:47:06.907126 ip-10-0-132-187 kubenswrapper[2577]: I0918 21:47:06.907097 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75f94bc69c-sgfsn_c438df90-6953-4453-8877-bbf1117bd8f7/copy-manifests/0.log"