Sep 10 14:47:41.071109 ip-10-0-143-4 systemd[1]: Starting Kubernetes Kubelet... Sep 10 14:47:41.479961 ip-10-0-143-4 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:47:41.479961 ip-10-0-143-4 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 14:47:41.479961 ip-10-0-143-4 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:47:41.479961 ip-10-0-143-4 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 14:47:41.479961 ip-10-0-143-4 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:47:41.480584 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.479978 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 14:47:41.485953 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485935 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:41.485953 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485950 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:41.485953 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485954 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485959 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485962 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485965 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485968 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485971 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485974 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485981 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485984 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485987 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485989 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485992 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485994 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.485999 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486003 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486005 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486008 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486010 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486013 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486015 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:41.486051 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486018 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486022 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486026 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486029 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486032 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486034 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486037 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486040 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486043 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486045 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486048 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486051 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486053 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486056 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486059 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486062 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486064 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486067 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486071 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:41.486531 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486074 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486077 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486079 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486082 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486084 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486087 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486089 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486092 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486094 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486097 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486099 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486102 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486104 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486106 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486109 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486112 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486115 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486118 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486120 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:41.487010 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486123 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486126 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486128 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486131 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486134 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486137 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486141 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486144 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486146 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486149 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486152 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486154 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486157 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486159 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486162 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486165 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486167 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486171 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486173 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486176 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:41.487478 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486178 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486181 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486183 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486186 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486188 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486191 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486193 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486196 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486199 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486202 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486204 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486207 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486210 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486212 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486215 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486217 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486220 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486223 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:41.487970 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.486225 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:41.488403 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488381 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:41.488403 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488388 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:41.488403 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488391 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:41.488403 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488394 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:41.488403 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488397 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:41.488403 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488400 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:41.488403 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488403 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:41.488403 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488406 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488409 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488412 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488415 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488417 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488420 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488422 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488425 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488428 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488430 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488432 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488435 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488437 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488440 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488443 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488445 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488448 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488450 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488453 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488455 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:41.488605 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488458 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488461 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488463 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488466 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488468 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488470 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488473 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488476 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488478 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488481 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488483 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488486 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488488 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488491 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488497 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488502 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488504 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488508 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488511 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:41.489130 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488514 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488517 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488520 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488523 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488525 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488528 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488530 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488533 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488535 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488538 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488542 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488544 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488547 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488549 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488552 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488554 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488557 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488560 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488562 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488565 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:41.489940 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488567 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488570 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488573 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488577 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488580 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488584 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488586 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488589 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488592 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488595 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488598 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488600 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488603 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488605 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488608 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488611 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488613 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488616 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488619 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:41.490597 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488621 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488624 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488626 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488629 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488632 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488634 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488636 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488639 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488641 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488646 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488649 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488652 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488655 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.488658 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490424 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490437 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490441 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490444 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490447 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:41.491118 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490450 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490454 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490457 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490460 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490462 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490465 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490468 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490471 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490474 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490477 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490480 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490482 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490485 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490488 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490491 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490494 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490496 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490499 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490502 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:41.491604 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490504 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490509 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490513 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490516 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490519 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490522 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490524 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490527 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490530 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490533 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490537 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490540 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490543 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490545 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490548 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490551 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490554 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490556 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490559 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490562 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:41.492078 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490565 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490567 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490570 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490572 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490575 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490578 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490583 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490585 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490588 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490591 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490594 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490596 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490599 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490603 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490608 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490611 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490614 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490617 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490620 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490624 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:41.492561 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490627 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490630 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490633 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490635 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490638 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490641 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490644 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490646 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490649 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490651 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490654 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490657 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490659 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490662 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490664 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490667 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490671 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490675 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490677 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:41.493103 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490680 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490683 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490686 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490688 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490691 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490693 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490696 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490699 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490701 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490704 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490706 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490709 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490712 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490714 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490718 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.490720 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490817 2560 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490851 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490858 2560 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490863 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490867 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 14:47:41.493555 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490870 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490874 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490879 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490882 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490885 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490888 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490892 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490895 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490898 2560 flags.go:64] FLAG: --cgroup-root="" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490901 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490904 2560 flags.go:64] FLAG: --client-ca-file="" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490906 2560 flags.go:64] FLAG: --cloud-provider="external" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490911 2560 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490916 2560 flags.go:64] FLAG: --cluster-domain="" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490919 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490922 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490925 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490928 2560 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490932 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490934 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490937 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490941 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490944 2560 flags.go:64] FLAG: --contention-profiling="false" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490947 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 14:47:41.494149 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490950 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490954 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490957 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490961 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490964 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490967 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490971 2560 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490974 2560 flags.go:64] FLAG: --enable-server="true" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490977 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490983 2560 flags.go:64] FLAG: --event-burst="100" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490986 2560 flags.go:64] FLAG: --event-qps="50" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490989 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490992 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490995 2560 flags.go:64] FLAG: --eviction-hard="" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.490999 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491002 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491005 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491008 2560 flags.go:64] FLAG: --eviction-soft="" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491011 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491014 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491016 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491020 2560 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491023 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491026 2560 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491029 2560 flags.go:64] FLAG: --feature-gates="" Sep 10 14:47:41.494874 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491033 2560 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491036 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491040 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491043 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491046 2560 flags.go:64] FLAG: --healthz-port="10248" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491049 2560 flags.go:64] FLAG: --help="false" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491052 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491055 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491058 2560 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491061 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491065 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491068 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491071 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491074 2560 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491076 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491079 2560 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491082 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491085 2560 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491088 2560 flags.go:64] FLAG: --kube-reserved="" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491091 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491094 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491097 2560 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491100 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491102 2560 flags.go:64] FLAG: --lock-file="" Sep 10 14:47:41.495492 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491105 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491108 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491111 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491116 2560 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491119 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491123 2560 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491126 2560 flags.go:64] FLAG: --logging-format="text" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491129 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491133 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491139 2560 flags.go:64] FLAG: --manifest-url="" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491141 2560 flags.go:64] FLAG: --manifest-url-header="" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491146 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491149 2560 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491153 2560 flags.go:64] FLAG: --max-pods="110" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491156 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491159 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491163 2560 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491166 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491169 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491172 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491175 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491183 2560 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491186 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491190 2560 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 14:47:41.496101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491193 2560 flags.go:64] FLAG: --pod-cidr="" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491196 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491201 2560 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491204 2560 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491207 2560 flags.go:64] FLAG: --pods-per-core="0" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491210 2560 flags.go:64] FLAG: --port="10250" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491213 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491216 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a5bc886eaeb8618" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491219 2560 flags.go:64] FLAG: --qos-reserved="" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491222 2560 flags.go:64] FLAG: --read-only-port="10255" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491225 2560 flags.go:64] FLAG: --register-node="true" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491228 2560 flags.go:64] FLAG: --register-with-taints="" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491232 2560 flags.go:64] FLAG: --registry-burst="10" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491235 2560 flags.go:64] FLAG: --registry-qps="5" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491239 2560 flags.go:64] FLAG: --reserved-cpus="" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491242 2560 flags.go:64] FLAG: --reserved-memory="" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491246 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491249 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491253 2560 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491256 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491260 2560 flags.go:64] FLAG: --runonce="false" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491263 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491266 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491269 2560 flags.go:64] FLAG: --seccomp-default="false" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491272 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 14:47:41.496674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491276 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491279 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491282 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491285 2560 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491288 2560 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491291 2560 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491294 2560 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491297 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491300 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491303 2560 flags.go:64] FLAG: --system-cgroups="" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491306 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491312 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491315 2560 flags.go:64] FLAG: --tls-cert-file="" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491318 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491322 2560 flags.go:64] FLAG: --tls-min-version="" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491325 2560 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491328 2560 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491331 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491334 2560 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491337 2560 flags.go:64] FLAG: --v="2" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491341 2560 flags.go:64] FLAG: --version="false" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491345 2560 flags.go:64] FLAG: --vmodule="" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491349 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491352 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491463 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:41.497318 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491468 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491471 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491474 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491476 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491479 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491483 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491486 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491489 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491491 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491494 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491496 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491499 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491502 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491504 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491507 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491510 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491512 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491515 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491517 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491520 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:41.497987 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491523 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491526 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491528 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491531 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491534 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491536 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491539 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491542 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491544 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491547 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491551 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491554 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491557 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491560 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491563 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491565 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491568 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491572 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491574 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491577 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:41.498476 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491580 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491582 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491585 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491587 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491590 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491592 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491595 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491598 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491600 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491603 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491606 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491608 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491611 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491614 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491617 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491619 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491626 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491630 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491633 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:41.499082 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491637 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491639 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491642 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491644 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491648 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491651 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491654 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491657 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491659 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491662 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491666 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491668 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491671 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491674 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491676 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491679 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491681 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491686 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491689 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:41.499539 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491692 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491695 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491698 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491700 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491714 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491718 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491721 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491724 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491727 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491730 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491733 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491736 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491739 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491742 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491744 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491747 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491749 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491752 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491756 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:41.500017 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.491759 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.491764 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.499010 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.499025 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.499706 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.499714 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499771 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499776 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499779 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499782 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499785 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499788 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499792 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499812 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499815 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:41.500487 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499818 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499820 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499823 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499825 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499828 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499830 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499833 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499836 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499839 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499841 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499844 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499846 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499849 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499851 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499854 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499856 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499859 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499861 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499864 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499866 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:41.500878 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499871 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499874 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499877 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499881 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499884 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499888 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499891 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499894 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499897 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499900 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499903 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499906 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499909 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499912 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499914 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499917 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499920 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499923 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499926 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499929 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:41.501388 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499932 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499934 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499938 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499942 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499945 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499948 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499951 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499954 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499956 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499959 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499962 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499964 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499968 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499972 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499974 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499977 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499980 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499983 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499985 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:41.501901 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499988 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499990 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499993 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499995 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.499998 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500001 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500003 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500006 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500008 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500011 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500013 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500016 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500019 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500022 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500024 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500027 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500030 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500033 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500035 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500038 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:41.502361 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500040 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500043 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500045 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500048 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500051 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500054 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500056 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500060 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500062 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500065 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500067 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.500072 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500177 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500181 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500185 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:47:41.502887 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500187 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500190 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500193 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500196 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500199 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500202 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500205 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500208 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500211 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500213 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500216 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500219 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500221 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500224 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500227 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500229 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500232 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500234 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500237 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500240 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:47:41.503271 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500242 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500245 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500247 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500251 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500255 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500258 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500261 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500264 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500266 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500269 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500271 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500274 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500277 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500279 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500282 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500284 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500287 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500289 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500292 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:47:41.503778 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500295 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500297 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500300 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500303 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500305 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500308 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500310 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500313 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500315 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500318 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500320 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500323 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500325 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500328 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500330 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500333 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500336 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500338 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500341 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500344 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:47:41.504347 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500346 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500349 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500351 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500354 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500356 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500359 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500362 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500364 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500367 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500369 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500372 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500375 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500377 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500380 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500383 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500385 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500388 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500391 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500393 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500396 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:47:41.504856 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500399 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500401 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500404 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500406 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500409 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500411 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500414 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500418 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500422 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500425 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500429 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500432 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500435 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500438 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500441 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500444 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:47:41.505419 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:41.500447 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:47:41.505854 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.500451 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:47:41.505854 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.500550 2560 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 14:47:41.505854 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.503161 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 14:47:41.505854 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.504047 2560 server.go:1019] "Starting client certificate rotation" Sep 10 14:47:41.505854 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.504168 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 14:47:41.505854 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.504219 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 14:47:41.507267 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.507251 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 14:47:41.508343 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.508322 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 14:47:41.518870 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.518851 2560 log.go:25] "Validated CRI v1 runtime API" Sep 10 14:47:41.521779 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.521762 2560 log.go:25] "Validated CRI v1 image API" Sep 10 14:47:41.523263 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.523250 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 14:47:41.528318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.528156 2560 fs.go:135] Filesystem UUIDs: map[24b06a9b-7225-4d22-b876-9dee8ebe7cf0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b37f6fc1-5837-4bd0-8458-de8511e3b8b1:/dev/nvme0n1p4] Sep 10 14:47:41.528318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.528182 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 14:47:41.534312 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.534201 2560 manager.go:216] Machine: {Timestamp:2026-09-10 14:47:41.532867272 +0000 UTC m=+0.347450013 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099886 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f540bf209727374f285d518e9d4c8 SystemUUID:ec2f540b-f209-7273-74f2-85d518e9d4c8 BootID:54710f58-f7e6-46ee-bb23-28ab96f877a7 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4b:9a:b9:35:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4b:9a:b9:35:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:25:1b:8b:9d:d8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 14:47:41.534312 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.534304 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 14:47:41.534423 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.534374 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 14:47:41.535461 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.535444 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 14:47:41.537089 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.537066 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 14:47:41.537219 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.537093 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 14:47:41.538540 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.538530 2560 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 14:47:41.538577 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.538543 2560 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 14:47:41.538577 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.538551 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 14:47:41.539254 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.539244 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 14:47:41.540556 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.540547 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 14:47:41.544543 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.544532 2560 state_mem.go:36] "Initialized new in-memory state store" Sep 10 14:47:41.544661 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.544652 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 14:47:41.544689 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.544680 2560 kubelet.go:486] "Attempting to sync node with API server" Sep 10 14:47:41.544724 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.544692 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 14:47:41.544724 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.544715 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 14:47:41.544780 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.544730 2560 kubelet.go:398] "Adding apiserver pod source" Sep 10 14:47:41.544780 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.544741 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 14:47:41.545058 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.545046 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 14:47:41.545108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.545064 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 14:47:41.547358 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.547342 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 14:47:41.549700 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.549687 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 14:47:41.549756 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.549717 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 14:47:41.549894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.549884 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 14:47:41.549925 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.549899 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 14:47:41.549925 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.549906 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 14:47:41.549925 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.549912 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 14:47:41.549925 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.549920 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 14:47:41.549925 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.549926 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 14:47:41.550561 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.550551 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 14:47:41.550561 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.550562 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 14:47:41.550622 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.550568 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 14:47:41.550622 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.550575 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 14:47:41.550622 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.550587 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 14:47:41.550622 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.550595 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 14:47:41.550622 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.550620 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 14:47:41.550764 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.550625 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 14:47:41.554102 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.554088 2560 server.go:1268] "Started kubelet" Sep 10 14:47:41.554179 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.554145 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 14:47:41.554333 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.554284 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 14:47:41.554390 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.554356 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 14:47:41.555159 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.555141 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 14:47:41.555265 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.555249 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 14:47:41.555324 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.555307 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 14:47:41.555503 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.555487 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 14:47:41.556237 ip-10-0-143-4 systemd[1]: Started Kubernetes Kubelet. Sep 10 14:47:41.558309 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.558165 2560 server.go:310] "Adding debug handlers to kubelet server" Sep 10 14:47:41.561174 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.560140 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-4.ec2.internal.18d3fd01e3c0893e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-4.ec2.internal,UID:ip-10-0-143-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-4.ec2.internal,},FirstTimestamp:2026-09-10 14:47:41.554059582 +0000 UTC m=+0.368642324,LastTimestamp:2026-09-10 14:47:41.554059582 +0000 UTC m=+0.368642324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-4.ec2.internal,}" Sep 10 14:47:41.561354 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.561329 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 14:47:41.561864 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.561846 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 14:47:41.561953 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.561907 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 14:47:41.562027 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.562003 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r55rf" Sep 10 14:47:41.562084 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.562064 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:41.562596 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.562579 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 14:47:41.562673 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.562660 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 14:47:41.562734 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.562712 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 14:47:41.562824 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.562786 2560 reconstruct.go:97] "Volume reconstruction finished" Sep 10 14:47:41.562890 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.562823 2560 reconciler.go:29] "Reconciler: start to sync state" Sep 10 14:47:41.563411 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.563393 2560 factory.go:55] Registering systemd factory Sep 10 14:47:41.563411 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.563411 2560 factory.go:223] Registration of the systemd container factory successfully Sep 10 14:47:41.563602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.563587 2560 factory.go:153] Registering CRI-O factory Sep 10 14:47:41.563602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.563602 2560 factory.go:223] Registration of the crio container factory successfully Sep 10 14:47:41.563682 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.563643 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 14:47:41.563682 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.563655 2560 factory.go:103] Registering Raw factory Sep 10 14:47:41.563682 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.563665 2560 manager.go:1196] Started watching for new ooms in manager Sep 10 14:47:41.564090 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.564068 2560 manager.go:349] Starting recovery of all containers Sep 10 14:47:41.564176 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.564103 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 14:47:41.564176 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.564133 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 14:47:41.567583 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.567557 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r55rf" Sep 10 14:47:41.575245 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.575230 2560 manager.go:354] Recovery completed Sep 10 14:47:41.579759 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.579745 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:41.581400 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.581385 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:41.581468 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.581417 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:41.581468 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.581434 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:41.582084 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.582066 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 14:47:41.582084 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.582084 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:41.582189 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.582085 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 14:47:41.582189 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.582179 2560 state_mem.go:36] "Initialized new in-memory state store" Sep 10 14:47:41.582845 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.582827 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:41.582922 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.582855 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:41.582922 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.582877 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:41.584335 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.584320 2560 policy_none.go:49] "None policy: Start" Sep 10 14:47:41.584421 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.584338 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 14:47:41.584421 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.584352 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 14:47:41.586611 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.586597 2560 policy_none.go:47] "Start" Sep 10 14:47:41.616430 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.616416 2560 manager.go:337] "Starting Device Plugin manager" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.616453 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.616465 2560 server.go:87] "Starting device plugin registration server" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.616751 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.616838 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.616938 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.617037 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.617049 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.617588 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 14:47:41.636962 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.617656 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:41.656182 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.656157 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 14:47:41.658767 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.657903 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 14:47:41.658767 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.657955 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 14:47:41.658767 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.657981 2560 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 14:47:41.658767 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.658028 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 14:47:41.660770 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.660749 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:41.717595 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.717569 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:41.718497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.718480 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:41.718598 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.718513 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:41.718598 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.718527 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:41.718598 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.718555 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.727273 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.727256 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.727357 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.727284 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-4.ec2.internal\": node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:41.744693 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.744624 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:41.759105 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.759082 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal"] Sep 10 14:47:41.759174 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.759143 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:41.759925 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.759910 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:41.759996 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.759939 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:41.759996 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.759949 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:41.762176 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.762164 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:41.762316 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.762302 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.762362 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.762333 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:41.762927 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.762910 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:41.763012 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.762911 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:41.763012 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.762957 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:41.763012 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.762969 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:41.763012 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.762936 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:41.763012 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.763009 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:41.764281 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.764268 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e982a9417947f5747d74445e3d9c6656-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal\" (UID: \"e982a9417947f5747d74445e3d9c6656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.764344 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.764290 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e982a9417947f5747d74445e3d9c6656-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal\" (UID: \"e982a9417947f5747d74445e3d9c6656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.764344 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.764307 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2320f504613112cbb2f124197a41f0d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-4.ec2.internal\" (UID: \"b2320f504613112cbb2f124197a41f0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.765103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.765089 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.765153 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.765114 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:47:41.765736 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.765710 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:47:41.765793 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.765744 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:47:41.765793 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.765760 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:47:41.784450 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.784425 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-4.ec2.internal\" not found" node="ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.788836 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.788822 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-4.ec2.internal\" not found" node="ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.845414 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.845395 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:41.865308 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.865293 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2320f504613112cbb2f124197a41f0d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-4.ec2.internal\" (UID: \"b2320f504613112cbb2f124197a41f0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.865382 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.865319 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e982a9417947f5747d74445e3d9c6656-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal\" (UID: \"e982a9417947f5747d74445e3d9c6656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.865382 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.865342 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e982a9417947f5747d74445e3d9c6656-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal\" (UID: \"e982a9417947f5747d74445e3d9c6656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.865382 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.865368 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e982a9417947f5747d74445e3d9c6656-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal\" (UID: \"e982a9417947f5747d74445e3d9c6656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.865491 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.865389 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2320f504613112cbb2f124197a41f0d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-4.ec2.internal\" (UID: \"b2320f504613112cbb2f124197a41f0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.865491 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:41.865398 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e982a9417947f5747d74445e3d9c6656-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal\" (UID: \"e982a9417947f5747d74445e3d9c6656\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:41.946360 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:41.946338 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.047187 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.047099 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.087638 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.087611 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:42.092845 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.092537 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" Sep 10 14:47:42.147349 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.147323 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.247888 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.247864 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.350328 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.350292 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.451887 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.451851 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.503910 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.503874 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 14:47:42.504335 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.504018 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 14:47:42.552763 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.552707 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.571754 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.571718 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 14:42:41 +0000 UTC" deadline="2028-03-10 14:06:33.93156908 +0000 UTC" Sep 10 14:47:42.571754 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.571751 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13127h18m51.35982068s" Sep 10 14:47:42.586450 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.586428 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:42.653652 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.653633 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.706852 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.706809 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:42.755984 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.755947 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.815027 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:42.814858 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2320f504613112cbb2f124197a41f0d.slice/crio-88b7e7d7fac48b43b1e6eed69fd77b842fb4f4f535ff1cbbfa198765ea4a57c0 WatchSource:0}: Error finding container 88b7e7d7fac48b43b1e6eed69fd77b842fb4f4f535ff1cbbfa198765ea4a57c0: Status 404 returned error can't find the container with id 88b7e7d7fac48b43b1e6eed69fd77b842fb4f4f535ff1cbbfa198765ea4a57c0 Sep 10 14:47:42.815304 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:42.815286 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode982a9417947f5747d74445e3d9c6656.slice/crio-07768f574c90385b67db2e1ef7873296608c7d6d14bc983f49ace457431819fe WatchSource:0}: Error finding container 07768f574c90385b67db2e1ef7873296608c7d6d14bc983f49ace457431819fe: Status 404 returned error can't find the container with id 07768f574c90385b67db2e1ef7873296608c7d6d14bc983f49ace457431819fe Sep 10 14:47:42.820575 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:42.820554 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:47:42.858086 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.858054 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:42.964827 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:42.964787 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:43.065920 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.065867 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-4.ec2.internal\" not found" Sep 10 14:47:43.140163 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.140139 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:43.163107 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.163085 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" Sep 10 14:47:43.173809 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.173778 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 14:47:43.175067 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.175053 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" Sep 10 14:47:43.184153 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.184136 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 14:47:43.545856 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.545783 2560 apiserver.go:52] "Watching apiserver" Sep 10 14:47:43.552639 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.552615 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 14:47:43.554842 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.554802 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-d2txf","kube-system/konnectivity-agent-p87s2","kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal","openshift-multus/multus-l65cl","openshift-network-operator/iptables-alerter-26xrt","openshift-cluster-node-tuning-operator/tuned-v62bw","openshift-dns/node-resolver-4r7d6","openshift-image-registry/node-ca-l298v","openshift-multus/multus-additional-cni-plugins-56k7w","openshift-multus/network-metrics-daemon-ts66m","openshift-network-diagnostics/network-check-target-msnqz","openshift-ovn-kubernetes/ovnkube-node-92lh7"] Sep 10 14:47:43.557789 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.557763 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.560066 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.560043 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.560527 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.560413 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.561194 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.561223 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.561712 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-49sb6\"" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.561762 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.562112 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.562448 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.562606 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.562802 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 14:47:43.562934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.562870 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vv7f\"" Sep 10 14:47:43.563576 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.563556 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 14:47:43.563650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.563614 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 14:47:43.564847 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.563709 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 14:47:43.564847 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.563911 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 14:47:43.565314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.565292 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.567588 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.567564 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 14:47:43.568588 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.568333 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 14:47:43.568664 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.568611 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:47:43.568753 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.568613 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qn8b9\"" Sep 10 14:47:43.569018 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.568976 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.571329 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.571308 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.571682 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.571658 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 14:47:43.571858 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.571748 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zsrsg\"" Sep 10 14:47:43.571858 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.571658 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 14:47:43.571858 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.571661 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 14:47:43.573744 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.573718 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 14:47:43.574081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.574057 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jz7xn\"" Sep 10 14:47:43.574170 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.574079 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 14:47:43.574334 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.574316 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.574531 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.574508 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:43.576542 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576517 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-log-socket\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.576640 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576554 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-cni-bin\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.576640 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576581 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovnkube-config\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.576640 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576608 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-slash\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.576803 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576639 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.576803 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576666 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.576803 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576690 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-ovn\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.576803 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576714 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-env-overrides\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.576803 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576756 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf4m5\" (UniqueName: \"kubernetes.io/projected/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-kube-api-access-tf4m5\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576822 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-system-cni-dir\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576871 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-run-ovn-kubernetes\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576896 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovn-node-metrics-cert\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576924 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-host-slash\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576950 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-tuning-conf-dir\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576973 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-cni-netd\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.576996 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovnkube-script-lib\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577020 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkdx\" (UniqueName: \"kubernetes.io/projected/9ccb0608-8ad0-48d7-bb71-3f258e830b99-kube-api-access-zlkdx\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577103 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577079 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-systemd\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577110 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-iptables-alerter-script\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577139 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-cnibin\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577164 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-cni-binary-copy\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577189 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-systemd-units\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577241 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577268 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-kubelet\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577436 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-etc-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577475 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577501 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flq6g\" (UniqueName: \"kubernetes.io/projected/744583dc-6c66-45ed-9532-cd6950609908-kube-api-access-flq6g\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.577546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577534 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-var-lib-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.578179 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577558 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-node-log\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.578179 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.577583 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-run-netns\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.578282 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.578173 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.580637 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.580617 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:47:43.580765 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.580746 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hs8qf\"" Sep 10 14:47:43.580940 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.580623 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 14:47:43.583095 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.583080 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.585949 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.585929 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.586595 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.586393 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 14:47:43.586595 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.586473 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxl9b\"" Sep 10 14:47:43.586595 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.586531 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 14:47:43.588618 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.588603 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:43.588763 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.588742 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:47:43.589031 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.589011 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 14:47:43.589162 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.589141 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fgjfj\"" Sep 10 14:47:43.589239 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.589226 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 14:47:43.589713 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.589248 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 14:47:43.591233 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.591216 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:43.591338 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.591295 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:43.593750 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.593731 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:47:43.594593 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.594570 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mnzht" Sep 10 14:47:43.596647 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.596625 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 14:47:43.596746 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.596684 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mx8l5\"" Sep 10 14:47:43.596886 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.596868 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 14:47:43.607608 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.607587 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mnzht" Sep 10 14:47:43.663622 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.663597 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 14:47:43.664307 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.664215 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" event={"ID":"b2320f504613112cbb2f124197a41f0d","Type":"ContainerStarted","Data":"88b7e7d7fac48b43b1e6eed69fd77b842fb4f4f535ff1cbbfa198765ea4a57c0"} Sep 10 14:47:43.666062 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.666013 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" event={"ID":"e982a9417947f5747d74445e3d9c6656","Type":"ContainerStarted","Data":"07768f574c90385b67db2e1ef7873296608c7d6d14bc983f49ace457431819fe"} Sep 10 14:47:43.678256 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678229 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.678379 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678262 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-ovn\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.678379 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678289 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-netns\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.678379 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678319 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llld\" (UniqueName: \"kubernetes.io/projected/960656d9-d9fa-4953-bf65-f475c466ea0c-kube-api-access-9llld\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.678379 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678340 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l54c2\" (UniqueName: \"kubernetes.io/projected/a3660574-2df4-41f8-a2f4-bba32b7d76d4-kube-api-access-l54c2\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.678602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678391 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-ovn\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.678602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678452 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgxk\" (UniqueName: \"kubernetes.io/projected/c30f901a-49df-4c7f-a362-4b0ce6f76057-kube-api-access-kjgxk\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:43.678602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678488 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-system-cni-dir\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.678602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678518 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-daemon-config\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.678602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678555 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20b78c3e-ce14-498b-bdbf-8baa9718867f-hosts-file\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.678602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678572 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-system-cni-dir\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678580 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnq66\" (UniqueName: \"kubernetes.io/projected/20b78c3e-ce14-498b-bdbf-8baa9718867f-kube-api-access-qnq66\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678642 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/549f4770-f5a2-49d8-a3ce-d62df642b6e7-dbus\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678667 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysconfig\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678695 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-tuning-conf-dir\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678717 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-cni-netd\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678740 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovnkube-script-lib\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678764 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysctl-conf\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678794 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9425s\" (UniqueName: \"kubernetes.io/projected/a8eb91f7-476a-4c3a-b163-adac5bc16180-kube-api-access-9425s\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678833 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/549f4770-f5a2-49d8-a3ce-d62df642b6e7-kubelet-config\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678842 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-cni-netd\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678919 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-systemd\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678953 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-iptables-alerter-script\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.678983 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.678982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-modprobe-d\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679010 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-cnibin\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679036 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-systemd-units\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679062 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-conf-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-kubernetes\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679111 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed237cec-68e7-468d-ba17-ac16ced33017-konnectivity-ca\") pod \"konnectivity-agent-p87s2\" (UID: \"ed237cec-68e7-468d-ba17-ac16ced33017\") " pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679135 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-cnibin\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679159 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-cni-bin\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679183 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-hostroot\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679254 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3660574-2df4-41f8-a2f4-bba32b7d76d4-tmp\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-systemd-units\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679321 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-device-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679346 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-tuning-conf-dir\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679340 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-systemd\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679353 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679402 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed237cec-68e7-468d-ba17-ac16ced33017-agent-certs\") pod \"konnectivity-agent-p87s2\" (UID: \"ed237cec-68e7-468d-ba17-ac16ced33017\") " pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679418 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/744583dc-6c66-45ed-9532-cd6950609908-cnibin\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.679668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679447 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-systemd\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679455 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovnkube-script-lib\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679513 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovnkube-config\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679539 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysctl-d\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679685 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-iptables-alerter-script\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679747 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-tuned\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679775 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-registration-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679814 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-slash\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679819 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679894 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-env-overrides\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679902 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-run-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679914 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4m5\" (UniqueName: \"kubernetes.io/projected/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-kube-api-access-tf4m5\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679948 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-lib-modules\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679953 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.679973 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8eb91f7-476a-4c3a-b163-adac5bc16180-host\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.680650 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680003 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bpth\" (UniqueName: \"kubernetes.io/projected/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-kube-api-access-6bpth\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680013 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-slash\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680034 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovnkube-config\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680040 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-run-ovn-kubernetes\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680069 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovn-node-metrics-cert\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680097 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-run-ovn-kubernetes\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680147 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-host-slash\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680186 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680217 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-etc-kubernetes\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680249 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-run\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680275 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-socket-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680300 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-sys-fs\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680328 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkdx\" (UniqueName: \"kubernetes.io/projected/9ccb0608-8ad0-48d7-bb71-3f258e830b99-kube-api-access-zlkdx\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680355 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-cni-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680357 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-host-slash\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680361 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ccb0608-8ad0-48d7-bb71-3f258e830b99-env-overrides\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680387 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/960656d9-d9fa-4953-bf65-f475c466ea0c-cni-binary-copy\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.681534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680541 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-k8s-cni-cncf-io\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680572 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-socket-dir-parent\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680609 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-host\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680635 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680679 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-cni-binary-copy\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680716 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680734 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8eb91f7-476a-4c3a-b163-adac5bc16180-serviceca\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-kubelet\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680780 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-etc-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680787 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680809 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-cni-multus\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680838 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680874 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flq6g\" (UniqueName: \"kubernetes.io/projected/744583dc-6c66-45ed-9532-cd6950609908-kube-api-access-flq6g\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-var-lib-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680934 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-node-log\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-system-cni-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.680994 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-kubelet\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.682437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681020 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-multus-certs\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681044 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-sys\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681076 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-var-lib-kubelet\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681103 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-run-netns\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681129 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-log-socket\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681153 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-cni-bin\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b78c3e-ce14-498b-bdbf-8baa9718867f-tmp-dir\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681191 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-node-log\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681204 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681293 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-var-lib-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681330 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/744583dc-6c66-45ed-9532-cd6950609908-cni-binary-copy\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681341 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-log-socket\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681393 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-run-netns\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681431 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-etc-openvswitch\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681443 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-cni-bin\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.683380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.681440 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ccb0608-8ad0-48d7-bb71-3f258e830b99-host-kubelet\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.684436 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.684415 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:47:43.685667 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.685645 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ccb0608-8ad0-48d7-bb71-3f258e830b99-ovn-node-metrics-cert\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.689681 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.689658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4m5\" (UniqueName: \"kubernetes.io/projected/8bfd968c-b1e5-4296-92b7-02f1e672fb2f-kube-api-access-tf4m5\") pod \"iptables-alerter-26xrt\" (UID: \"8bfd968c-b1e5-4296-92b7-02f1e672fb2f\") " pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.691084 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.691057 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flq6g\" (UniqueName: \"kubernetes.io/projected/744583dc-6c66-45ed-9532-cd6950609908-kube-api-access-flq6g\") pod \"multus-additional-cni-plugins-56k7w\" (UID: \"744583dc-6c66-45ed-9532-cd6950609908\") " pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.691161 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.691108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkdx\" (UniqueName: \"kubernetes.io/projected/9ccb0608-8ad0-48d7-bb71-3f258e830b99-kube-api-access-zlkdx\") pod \"ovnkube-node-92lh7\" (UID: \"9ccb0608-8ad0-48d7-bb71-3f258e830b99\") " pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.782358 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782320 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8eb91f7-476a-4c3a-b163-adac5bc16180-serviceca\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.782502 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782366 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-cni-multus\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782502 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782386 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-system-cni-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782502 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782458 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-cni-multus\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782502 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782484 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-system-cni-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782519 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-kubelet\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782545 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-multus-certs\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782579 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-sys\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782625 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-multus-certs\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782630 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-kubelet\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782654 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-var-lib-kubelet\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782678 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b78c3e-ce14-498b-bdbf-8baa9718867f-tmp-dir\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782693 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782704 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-var-lib-kubelet\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782717 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-netns\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.782742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llld\" (UniqueName: \"kubernetes.io/projected/960656d9-d9fa-4953-bf65-f475c466ea0c-kube-api-access-9llld\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782764 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l54c2\" (UniqueName: \"kubernetes.io/projected/a3660574-2df4-41f8-a2f4-bba32b7d76d4-kube-api-access-l54c2\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgxk\" (UniqueName: \"kubernetes.io/projected/c30f901a-49df-4c7f-a362-4b0ce6f76057-kube-api-access-kjgxk\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782829 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-daemon-config\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782830 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782636 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-sys\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782856 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20b78c3e-ce14-498b-bdbf-8baa9718867f-hosts-file\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782881 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnq66\" (UniqueName: \"kubernetes.io/projected/20b78c3e-ce14-498b-bdbf-8baa9718867f-kube-api-access-qnq66\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782882 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8eb91f7-476a-4c3a-b163-adac5bc16180-serviceca\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782905 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/549f4770-f5a2-49d8-a3ce-d62df642b6e7-dbus\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysconfig\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782956 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20b78c3e-ce14-498b-bdbf-8baa9718867f-hosts-file\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782960 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysctl-conf\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.782986 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9425s\" (UniqueName: \"kubernetes.io/projected/a8eb91f7-476a-4c3a-b163-adac5bc16180-kube-api-access-9425s\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/549f4770-f5a2-49d8-a3ce-d62df642b6e7-kubelet-config\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783029 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b78c3e-ce14-498b-bdbf-8baa9718867f-tmp-dir\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783037 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783065 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-modprobe-d\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.783306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783093 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-conf-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783112 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-netns\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783119 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-kubernetes\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783153 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed237cec-68e7-468d-ba17-ac16ced33017-konnectivity-ca\") pod \"konnectivity-agent-p87s2\" (UID: \"ed237cec-68e7-468d-ba17-ac16ced33017\") " pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783181 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-cnibin\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783185 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysctl-conf\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783195 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysconfig\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783189 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/549f4770-f5a2-49d8-a3ce-d62df642b6e7-dbus\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783210 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-cni-bin\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783171 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-kubernetes\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783235 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-hostroot\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.783241 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3660574-2df4-41f8-a2f4-bba32b7d76d4-tmp\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.783288 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret podName:549f4770-f5a2-49d8-a3ce-d62df642b6e7 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:44.283272214 +0000 UTC m=+3.097854944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret") pod "global-pull-secret-syncer-d2txf" (UID: "549f4770-f5a2-49d8-a3ce-d62df642b6e7") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783290 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/549f4770-f5a2-49d8-a3ce-d62df642b6e7-kubelet-config\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783303 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-device-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783309 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-modprobe-d\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.784223 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783324 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed237cec-68e7-468d-ba17-ac16ced33017-agent-certs\") pod \"konnectivity-agent-p87s2\" (UID: \"ed237cec-68e7-468d-ba17-ac16ced33017\") " pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783341 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-systemd\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783351 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-conf-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783385 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-device-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.783433 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.783492 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs podName:c30f901a-49df-4c7f-a362-4b0ce6f76057 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:44.283475274 +0000 UTC m=+3.098058019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs") pod "network-metrics-daemon-ts66m" (UID: "c30f901a-49df-4c7f-a362-4b0ce6f76057") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783357 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783527 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-systemd\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783528 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysctl-d\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783577 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-tuned\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783591 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-cnibin\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-registration-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783638 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-lib-modules\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783663 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8eb91f7-476a-4c3a-b163-adac5bc16180-host\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783688 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bpth\" (UniqueName: \"kubernetes.io/projected/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-kube-api-access-6bpth\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783762 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed237cec-68e7-468d-ba17-ac16ced33017-konnectivity-ca\") pod \"konnectivity-agent-p87s2\" (UID: \"ed237cec-68e7-468d-ba17-ac16ced33017\") " pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783527 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-daemon-config\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783797 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-sysctl-d\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783839 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-registration-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783851 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-var-lib-cni-bin\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783903 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8eb91f7-476a-4c3a-b163-adac5bc16180-host\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783911 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-lib-modules\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783917 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-hostroot\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783941 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783971 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-etc-kubernetes\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.783996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-run\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784022 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-socket-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784020 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-etc-kubernetes\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784050 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-sys-fs\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784076 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-cni-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/960656d9-d9fa-4953-bf65-f475c466ea0c-cni-binary-copy\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784126 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-k8s-cni-cncf-io\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784144 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-socket-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784151 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-socket-dir-parent\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784157 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-cni-dir\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.785972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784090 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-run\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784185 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-host\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784210 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784214 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-host-run-k8s-cni-cncf-io\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784225 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-sys-fs\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784296 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3660574-2df4-41f8-a2f4-bba32b7d76d4-host\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784299 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/960656d9-d9fa-4953-bf65-f475c466ea0c-multus-socket-dir-parent\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784344 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.784601 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/960656d9-d9fa-4953-bf65-f475c466ea0c-cni-binary-copy\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.786153 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a3660574-2df4-41f8-a2f4-bba32b7d76d4-etc-tuned\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.786375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3660574-2df4-41f8-a2f4-bba32b7d76d4-tmp\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.786672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.786452 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed237cec-68e7-468d-ba17-ac16ced33017-agent-certs\") pod \"konnectivity-agent-p87s2\" (UID: \"ed237cec-68e7-468d-ba17-ac16ced33017\") " pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:47:43.791426 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.791402 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:43.791426 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.791428 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:43.791616 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.791442 2560 projected.go:196] Error preparing data for projected volume kube-api-access-v6vj9 for pod openshift-network-diagnostics/network-check-target-msnqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:43.791616 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:43.791505 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9 podName:f4d637a4-3174-4b3b-b117-1e531cba9054 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:44.291489548 +0000 UTC m=+3.106072301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v6vj9" (UniqueName: "kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9") pod "network-check-target-msnqz" (UID: "f4d637a4-3174-4b3b-b117-1e531cba9054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:43.792793 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.792769 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnq66\" (UniqueName: \"kubernetes.io/projected/20b78c3e-ce14-498b-bdbf-8baa9718867f-kube-api-access-qnq66\") pod \"node-resolver-4r7d6\" (UID: \"20b78c3e-ce14-498b-bdbf-8baa9718867f\") " pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.792931 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.792896 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9425s\" (UniqueName: \"kubernetes.io/projected/a8eb91f7-476a-4c3a-b163-adac5bc16180-kube-api-access-9425s\") pod \"node-ca-l298v\" (UID: \"a8eb91f7-476a-4c3a-b163-adac5bc16180\") " pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.793415 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.793390 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l54c2\" (UniqueName: \"kubernetes.io/projected/a3660574-2df4-41f8-a2f4-bba32b7d76d4-kube-api-access-l54c2\") pod \"tuned-v62bw\" (UID: \"a3660574-2df4-41f8-a2f4-bba32b7d76d4\") " pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.793759 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.793739 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llld\" (UniqueName: \"kubernetes.io/projected/960656d9-d9fa-4953-bf65-f475c466ea0c-kube-api-access-9llld\") pod \"multus-l65cl\" (UID: \"960656d9-d9fa-4953-bf65-f475c466ea0c\") " pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.794083 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.794059 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bpth\" (UniqueName: \"kubernetes.io/projected/fc7a5c70-b722-4d87-8fa0-7caa39ffa412-kube-api-access-6bpth\") pod \"aws-ebs-csi-driver-node-xlbr4\" (UID: \"fc7a5c70-b722-4d87-8fa0-7caa39ffa412\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.794483 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.794466 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgxk\" (UniqueName: \"kubernetes.io/projected/c30f901a-49df-4c7f-a362-4b0ce6f76057-kube-api-access-kjgxk\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:43.873087 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.873002 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-56k7w" Sep 10 14:47:43.882687 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.882642 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:47:43.894647 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.894613 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-26xrt" Sep 10 14:47:43.900658 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.900623 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" Sep 10 14:47:43.908881 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.908848 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l65cl" Sep 10 14:47:43.916830 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.916791 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v62bw" Sep 10 14:47:43.927640 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.927606 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4r7d6" Sep 10 14:47:43.934407 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.934387 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l298v" Sep 10 14:47:43.941202 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:43.941174 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:47:44.289319 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:44.289237 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:44.289319 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:44.289289 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:44.289535 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:44.289395 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:44.289535 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:44.289423 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:44.289535 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:44.289470 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret podName:549f4770-f5a2-49d8-a3ce-d62df642b6e7 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:45.289450727 +0000 UTC m=+4.104033459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret") pod "global-pull-secret-syncer-d2txf" (UID: "549f4770-f5a2-49d8-a3ce-d62df642b6e7") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:44.289535 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:44.289493 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs podName:c30f901a-49df-4c7f-a362-4b0ce6f76057 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:45.289482433 +0000 UTC m=+4.104065164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs") pod "network-metrics-daemon-ts66m" (UID: "c30f901a-49df-4c7f-a362-4b0ce6f76057") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:44.390666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:44.390634 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:44.390843 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:44.390822 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:44.390918 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:44.390855 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:44.390918 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:44.390869 2560 projected.go:196] Error preparing data for projected volume kube-api-access-v6vj9 for pod openshift-network-diagnostics/network-check-target-msnqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:44.391015 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:44.390936 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9 podName:f4d637a4-3174-4b3b-b117-1e531cba9054 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:45.390914536 +0000 UTC m=+4.205497272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6vj9" (UniqueName: "kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9") pod "network-check-target-msnqz" (UID: "f4d637a4-3174-4b3b-b117-1e531cba9054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:44.611720 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:44.611608 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:42:43 +0000 UTC" deadline="2028-04-04 00:32:55.066270235 +0000 UTC" Sep 10 14:47:44.611720 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:44.611663 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13713h45m10.454611668s" Sep 10 14:47:44.807792 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.807759 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8eb91f7_476a_4c3a_b163_adac5bc16180.slice/crio-8ec4a0be5c37a5b195b2cb92e02a9b073a53a3534cb17728a1dce9191fc9a5be WatchSource:0}: Error finding container 8ec4a0be5c37a5b195b2cb92e02a9b073a53a3534cb17728a1dce9191fc9a5be: Status 404 returned error can't find the container with id 8ec4a0be5c37a5b195b2cb92e02a9b073a53a3534cb17728a1dce9191fc9a5be Sep 10 14:47:44.815188 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.815008 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960656d9_d9fa_4953_bf65_f475c466ea0c.slice/crio-ab681302731ea8c647f76b57bf5672abfc56a139077a31afeca4cbc1389ee276 WatchSource:0}: Error finding container ab681302731ea8c647f76b57bf5672abfc56a139077a31afeca4cbc1389ee276: Status 404 returned error can't find the container with id ab681302731ea8c647f76b57bf5672abfc56a139077a31afeca4cbc1389ee276 Sep 10 14:47:44.818860 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.818833 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded237cec_68e7_468d_ba17_ac16ced33017.slice/crio-82b630ff5986780205294519b98e4e357778fb06ace7607f31105b66e39ef4af WatchSource:0}: Error finding container 82b630ff5986780205294519b98e4e357778fb06ace7607f31105b66e39ef4af: Status 404 returned error can't find the container with id 82b630ff5986780205294519b98e4e357778fb06ace7607f31105b66e39ef4af Sep 10 14:47:44.819272 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.819255 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744583dc_6c66_45ed_9532_cd6950609908.slice/crio-ae2a31a0b538664d039aaed3650bd555712e59ff1aeb2fa8a31fd122d7799cf2 WatchSource:0}: Error finding container ae2a31a0b538664d039aaed3650bd555712e59ff1aeb2fa8a31fd122d7799cf2: Status 404 returned error can't find the container with id ae2a31a0b538664d039aaed3650bd555712e59ff1aeb2fa8a31fd122d7799cf2 Sep 10 14:47:44.820519 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.820492 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfd968c_b1e5_4296_92b7_02f1e672fb2f.slice/crio-8ad77d956ba06f7a5a9f9e2c0c840aa2700e152d07a9ccf8814e787b9372b5be WatchSource:0}: Error finding container 8ad77d956ba06f7a5a9f9e2c0c840aa2700e152d07a9ccf8814e787b9372b5be: Status 404 returned error can't find the container with id 8ad77d956ba06f7a5a9f9e2c0c840aa2700e152d07a9ccf8814e787b9372b5be Sep 10 14:47:44.821349 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.821325 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccb0608_8ad0_48d7_bb71_3f258e830b99.slice/crio-04cbd4409c6b4e5117e20e74f3a3b3de32a76e82b70f5e976276534c5c9500d1 WatchSource:0}: Error finding container 04cbd4409c6b4e5117e20e74f3a3b3de32a76e82b70f5e976276534c5c9500d1: Status 404 returned error can't find the container with id 04cbd4409c6b4e5117e20e74f3a3b3de32a76e82b70f5e976276534c5c9500d1 Sep 10 14:47:44.823772 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.822984 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7a5c70_b722_4d87_8fa0_7caa39ffa412.slice/crio-7b05c0d9182798214625ea9a1cceba30199d5219028f69fc4e642c2a83ce32a9 WatchSource:0}: Error finding container 7b05c0d9182798214625ea9a1cceba30199d5219028f69fc4e642c2a83ce32a9: Status 404 returned error can't find the container with id 7b05c0d9182798214625ea9a1cceba30199d5219028f69fc4e642c2a83ce32a9 Sep 10 14:47:44.824157 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.824113 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3660574_2df4_41f8_a2f4_bba32b7d76d4.slice/crio-63d3dc35fe988e33bf958adff1d359c1d1ad1ff0f552c8f5aec4f58390c3693d WatchSource:0}: Error finding container 63d3dc35fe988e33bf958adff1d359c1d1ad1ff0f552c8f5aec4f58390c3693d: Status 404 returned error can't find the container with id 63d3dc35fe988e33bf958adff1d359c1d1ad1ff0f552c8f5aec4f58390c3693d Sep 10 14:47:44.826286 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:47:44.825656 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b78c3e_ce14_498b_bdbf_8baa9718867f.slice/crio-8e3235bd191cf560dc06bdc5b722f711b5b0592ae8bce341a94e1dafaa1d867d WatchSource:0}: Error finding container 8e3235bd191cf560dc06bdc5b722f711b5b0592ae8bce341a94e1dafaa1d867d: Status 404 returned error can't find the container with id 8e3235bd191cf560dc06bdc5b722f711b5b0592ae8bce341a94e1dafaa1d867d Sep 10 14:47:45.296894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.296772 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:45.297050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.296899 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:45.297050 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.297017 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:45.297050 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.297021 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:45.297237 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.297083 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret podName:549f4770-f5a2-49d8-a3ce-d62df642b6e7 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:47.297063299 +0000 UTC m=+6.111646053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret") pod "global-pull-secret-syncer-d2txf" (UID: "549f4770-f5a2-49d8-a3ce-d62df642b6e7") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:45.297237 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.297105 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs podName:c30f901a-49df-4c7f-a362-4b0ce6f76057 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:47.297093525 +0000 UTC m=+6.111676260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs") pod "network-metrics-daemon-ts66m" (UID: "c30f901a-49df-4c7f-a362-4b0ce6f76057") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:45.397965 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.397922 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:45.398155 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.398128 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:45.398155 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.398152 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:45.398265 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.398166 2560 projected.go:196] Error preparing data for projected volume kube-api-access-v6vj9 for pod openshift-network-diagnostics/network-check-target-msnqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:45.398265 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.398229 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9 podName:f4d637a4-3174-4b3b-b117-1e531cba9054 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:47.398208833 +0000 UTC m=+6.212791569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6vj9" (UniqueName: "kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9") pod "network-check-target-msnqz" (UID: "f4d637a4-3174-4b3b-b117-1e531cba9054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:45.615054 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.614862 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:42:43 +0000 UTC" deadline="2028-05-18 19:22:42.250249024 +0000 UTC" Sep 10 14:47:45.615054 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.614942 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14788h34m56.635311606s" Sep 10 14:47:45.658986 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.658958 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:45.659145 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.659110 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:45.659145 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.658973 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:45.659279 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.658958 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:45.659279 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.659215 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:45.659378 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:45.659337 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:47:45.676479 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.676415 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-26xrt" event={"ID":"8bfd968c-b1e5-4296-92b7-02f1e672fb2f","Type":"ContainerStarted","Data":"8ad77d956ba06f7a5a9f9e2c0c840aa2700e152d07a9ccf8814e787b9372b5be"} Sep 10 14:47:45.678750 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.678670 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerStarted","Data":"ae2a31a0b538664d039aaed3650bd555712e59ff1aeb2fa8a31fd122d7799cf2"} Sep 10 14:47:45.684403 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.684283 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p87s2" event={"ID":"ed237cec-68e7-468d-ba17-ac16ced33017","Type":"ContainerStarted","Data":"82b630ff5986780205294519b98e4e357778fb06ace7607f31105b66e39ef4af"} Sep 10 14:47:45.691185 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.691110 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l65cl" event={"ID":"960656d9-d9fa-4953-bf65-f475c466ea0c","Type":"ContainerStarted","Data":"ab681302731ea8c647f76b57bf5672abfc56a139077a31afeca4cbc1389ee276"} Sep 10 14:47:45.699860 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.699833 2560 generic.go:356] "Generic (PLEG): container finished" podID="e982a9417947f5747d74445e3d9c6656" containerID="eafb8786ac7d251d72eb0db5585340e14d4cae101f33149ea1b9c3daf05b9ee9" exitCode=0 Sep 10 14:47:45.699961 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.699916 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" event={"ID":"e982a9417947f5747d74445e3d9c6656","Type":"ContainerDied","Data":"eafb8786ac7d251d72eb0db5585340e14d4cae101f33149ea1b9c3daf05b9ee9"} Sep 10 14:47:45.711776 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.711730 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v62bw" event={"ID":"a3660574-2df4-41f8-a2f4-bba32b7d76d4","Type":"ContainerStarted","Data":"63d3dc35fe988e33bf958adff1d359c1d1ad1ff0f552c8f5aec4f58390c3693d"} Sep 10 14:47:45.732948 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.732918 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" event={"ID":"fc7a5c70-b722-4d87-8fa0-7caa39ffa412","Type":"ContainerStarted","Data":"7b05c0d9182798214625ea9a1cceba30199d5219028f69fc4e642c2a83ce32a9"} Sep 10 14:47:45.738524 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.738499 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l298v" event={"ID":"a8eb91f7-476a-4c3a-b163-adac5bc16180","Type":"ContainerStarted","Data":"8ec4a0be5c37a5b195b2cb92e02a9b073a53a3534cb17728a1dce9191fc9a5be"} Sep 10 14:47:45.744855 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.744787 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" event={"ID":"b2320f504613112cbb2f124197a41f0d","Type":"ContainerStarted","Data":"c24dff32834bf2289c0960b643543c60237150b59588c170ec119eb5793a4ae6"} Sep 10 14:47:45.761719 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.761640 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4r7d6" event={"ID":"20b78c3e-ce14-498b-bdbf-8baa9718867f","Type":"ContainerStarted","Data":"8e3235bd191cf560dc06bdc5b722f711b5b0592ae8bce341a94e1dafaa1d867d"} Sep 10 14:47:45.770162 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:45.770075 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"04cbd4409c6b4e5117e20e74f3a3b3de32a76e82b70f5e976276534c5c9500d1"} Sep 10 14:47:46.786739 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:46.786483 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" event={"ID":"e982a9417947f5747d74445e3d9c6656","Type":"ContainerStarted","Data":"c255e4b7d81c2222721c14eb663055b30c289af9282c66e6650ec325ae4e318d"} Sep 10 14:47:46.803975 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:46.803907 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-4.ec2.internal" podStartSLOduration=3.803885726 podStartE2EDuration="3.803885726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:45.759681197 +0000 UTC m=+4.574263951" watchObservedRunningTime="2026-09-10 14:47:46.803885726 +0000 UTC m=+5.618468478" Sep 10 14:47:46.804150 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:46.804091 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-4.ec2.internal" podStartSLOduration=3.80408256 podStartE2EDuration="3.80408256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:46.803085951 +0000 UTC m=+5.617668702" watchObservedRunningTime="2026-09-10 14:47:46.80408256 +0000 UTC m=+5.618665312" Sep 10 14:47:47.317687 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:47.317645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:47.317898 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:47.317748 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:47.317898 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.317888 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:47.318022 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.317954 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret podName:549f4770-f5a2-49d8-a3ce-d62df642b6e7 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:51.317933664 +0000 UTC m=+10.132516398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret") pod "global-pull-secret-syncer-d2txf" (UID: "549f4770-f5a2-49d8-a3ce-d62df642b6e7") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:47.318382 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.318360 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:47.318475 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.318418 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs podName:c30f901a-49df-4c7f-a362-4b0ce6f76057 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:51.31840192 +0000 UTC m=+10.132984655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs") pod "network-metrics-daemon-ts66m" (UID: "c30f901a-49df-4c7f-a362-4b0ce6f76057") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:47.419010 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:47.418972 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:47.419181 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.419156 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:47.419181 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.419180 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:47.419296 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.419193 2560 projected.go:196] Error preparing data for projected volume kube-api-access-v6vj9 for pod openshift-network-diagnostics/network-check-target-msnqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:47.419296 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.419256 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9 podName:f4d637a4-3174-4b3b-b117-1e531cba9054 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:51.419238884 +0000 UTC m=+10.233821627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6vj9" (UniqueName: "kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9") pod "network-check-target-msnqz" (UID: "f4d637a4-3174-4b3b-b117-1e531cba9054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:47.658934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:47.658861 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:47.659084 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.658995 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:47.659405 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:47.659385 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:47.659512 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.659493 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:47:47.659641 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:47.659625 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:47.659718 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:47.659699 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:49.661363 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:49.661330 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:49.661868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:49.661378 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:49.661868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:49.661330 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:49.661868 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:49.661480 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:47:49.661868 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:49.661581 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:49.661868 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:49.661662 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:51.353461 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:51.353421 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:51.353932 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:51.353500 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:51.353932 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.353791 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:51.353932 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.353842 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:51.353932 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.353897 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret podName:549f4770-f5a2-49d8-a3ce-d62df642b6e7 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:59.353879513 +0000 UTC m=+18.168462245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret") pod "global-pull-secret-syncer-d2txf" (UID: "549f4770-f5a2-49d8-a3ce-d62df642b6e7") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:51.353932 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.353912 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs podName:c30f901a-49df-4c7f-a362-4b0ce6f76057 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:59.353905807 +0000 UTC m=+18.168488535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs") pod "network-metrics-daemon-ts66m" (UID: "c30f901a-49df-4c7f-a362-4b0ce6f76057") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:51.454702 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:51.454659 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:51.454876 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.454826 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:51.454876 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.454846 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:51.454876 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.454858 2560 projected.go:196] Error preparing data for projected volume kube-api-access-v6vj9 for pod openshift-network-diagnostics/network-check-target-msnqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:51.454998 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.454912 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9 podName:f4d637a4-3174-4b3b-b117-1e531cba9054 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:59.45489432 +0000 UTC m=+18.269477054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6vj9" (UniqueName: "kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9") pod "network-check-target-msnqz" (UID: "f4d637a4-3174-4b3b-b117-1e531cba9054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:51.659440 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:51.659355 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:51.659595 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.659475 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:51.659595 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:51.659554 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:51.659702 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.659637 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:47:51.660042 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:51.660018 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:51.660152 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:51.660117 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:53.658984 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:53.658871 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:53.658984 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:53.658895 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:53.659530 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:53.659006 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:53.659530 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:53.659042 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:53.659530 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:53.659146 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:53.659530 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:53.659235 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:47:55.658486 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:55.658449 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:55.658486 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:55.658476 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:55.658993 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:55.658455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:55.658993 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:55.658625 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:55.658993 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:55.658785 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:47:55.658993 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:55.658916 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:57.658933 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:57.658899 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:57.659379 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:57.658898 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:57.659379 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:57.659057 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:47:57.659379 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:57.659116 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:57.659379 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:57.659205 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:57.659379 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:57.659291 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:59.415637 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:59.415569 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:59.416100 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:59.415684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:59.416100 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.415735 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:59.416100 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.415786 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:59.416100 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.415831 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs podName:c30f901a-49df-4c7f-a362-4b0ce6f76057 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:15.415811169 +0000 UTC m=+34.230393912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs") pod "network-metrics-daemon-ts66m" (UID: "c30f901a-49df-4c7f-a362-4b0ce6f76057") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:47:59.416100 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.415851 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret podName:549f4770-f5a2-49d8-a3ce-d62df642b6e7 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:15.41584221 +0000 UTC m=+34.230424941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret") pod "global-pull-secret-syncer-d2txf" (UID: "549f4770-f5a2-49d8-a3ce-d62df642b6e7") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:47:59.516057 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:59.516026 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:59.516233 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.516169 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:47:59.516233 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.516188 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:47:59.516233 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.516200 2560 projected.go:196] Error preparing data for projected volume kube-api-access-v6vj9 for pod openshift-network-diagnostics/network-check-target-msnqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:59.516346 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.516260 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9 podName:f4d637a4-3174-4b3b-b117-1e531cba9054 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:15.516240486 +0000 UTC m=+34.330823215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6vj9" (UniqueName: "kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9") pod "network-check-target-msnqz" (UID: "f4d637a4-3174-4b3b-b117-1e531cba9054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:47:59.658813 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:59.658779 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:47:59.658963 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:59.658845 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:47:59.658963 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:47:59.658848 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:47:59.658963 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.658939 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:47:59.659156 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.659064 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:47:59.659211 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:47:59.659155 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:01.659060 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:01.659029 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:01.659491 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:01.659138 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:48:01.659491 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:01.659164 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:01.659491 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:01.659263 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:01.659491 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:01.659306 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:01.659491 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:01.659347 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:48:02.819749 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.819712 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4r7d6" event={"ID":"20b78c3e-ce14-498b-bdbf-8baa9718867f","Type":"ContainerStarted","Data":"fcdfe028540ad8b4c3f1dc68245b73b6159f002576064987cc96fd8b0210777d"} Sep 10 14:48:02.821847 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.821822 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:48:02.822251 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.822223 2560 generic.go:356] "Generic (PLEG): container finished" podID="9ccb0608-8ad0-48d7-bb71-3f258e830b99" containerID="3c0a488a384c2a07732a79f25788adf77a62ea0d203cc6ce4f97d801dfb2b127" exitCode=1 Sep 10 14:48:02.822456 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.822258 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerDied","Data":"3c0a488a384c2a07732a79f25788adf77a62ea0d203cc6ce4f97d801dfb2b127"} Sep 10 14:48:02.822456 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.822346 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"b529ec125ed1703b160c956c06b2da05ef06b146f9213fae6c3226b2424117e8"} Sep 10 14:48:02.824596 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.824568 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerStarted","Data":"30242a0e4d1ed6304cad7c820d617910302798469a936cf0f50884b4d1b718d9"} Sep 10 14:48:02.826682 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.826655 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p87s2" event={"ID":"ed237cec-68e7-468d-ba17-ac16ced33017","Type":"ContainerStarted","Data":"54de60b37859d4284dc702ff9cd6261e48cd423effacc2dd1f3ed6dcb80ceb60"} Sep 10 14:48:02.830548 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.830322 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v62bw" event={"ID":"a3660574-2df4-41f8-a2f4-bba32b7d76d4","Type":"ContainerStarted","Data":"3e89ed00790088232f9b3bc436ad321d4455e3f93240bf09b62e8918a9930e5c"} Sep 10 14:48:02.832215 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.832151 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" event={"ID":"fc7a5c70-b722-4d87-8fa0-7caa39ffa412","Type":"ContainerStarted","Data":"cca066caff661bf50f41fb2de7914e3fc30fd798d27d1f4151edff755138ffb2"} Sep 10 14:48:02.834218 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.833885 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l298v" event={"ID":"a8eb91f7-476a-4c3a-b163-adac5bc16180","Type":"ContainerStarted","Data":"b8cac31a766d972b18303b1d1504f69cf52640c1b283f36a130c5b106f035c4e"} Sep 10 14:48:02.839416 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.837557 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4r7d6" podStartSLOduration=3.20844078 podStartE2EDuration="20.837539298s" totalImagesPullingTime="17.629098518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:02.837165113 +0000 UTC m=+21.651747865" watchObservedRunningTime="2026-09-10 14:48:02.837539298 +0000 UTC m=+21.652122051" Sep 10 14:48:02.851693 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.851213 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l298v" podStartSLOduration=3.224103056 podStartE2EDuration="20.851196593s" totalImagesPullingTime="17.627093537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:02.850995262 +0000 UTC m=+21.665578015" watchObservedRunningTime="2026-09-10 14:48:02.851196593 +0000 UTC m=+21.665779344" Sep 10 14:48:02.868969 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.868873 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v62bw" podStartSLOduration=3.236618399 podStartE2EDuration="20.868858338s" totalImagesPullingTime="17.632239939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:02.868586386 +0000 UTC m=+21.683169139" watchObservedRunningTime="2026-09-10 14:48:02.868858338 +0000 UTC m=+21.683441088" Sep 10 14:48:02.884367 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:02.884325 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p87s2" podStartSLOduration=10.691878266 podStartE2EDuration="20.884311143s" totalImagesPullingTime="10.192432877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:02.884020316 +0000 UTC m=+21.698603065" watchObservedRunningTime="2026-09-10 14:48:02.884311143 +0000 UTC m=+21.698893893" Sep 10 14:48:03.659138 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.658955 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:03.659283 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.658960 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:03.659283 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:03.659205 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:48:03.659283 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.659014 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:03.659392 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:03.659278 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:48:03.659392 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:03.659368 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:03.839909 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.839604 2560 generic.go:356] "Generic (PLEG): container finished" podID="744583dc-6c66-45ed-9532-cd6950609908" containerID="30242a0e4d1ed6304cad7c820d617910302798469a936cf0f50884b4d1b718d9" exitCode=0 Sep 10 14:48:03.839909 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.839707 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerDied","Data":"30242a0e4d1ed6304cad7c820d617910302798469a936cf0f50884b4d1b718d9"} Sep 10 14:48:03.842169 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.842141 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l65cl" event={"ID":"960656d9-d9fa-4953-bf65-f475c466ea0c","Type":"ContainerStarted","Data":"68713b6ca9b655ee3799083c6c26f832e15d6e9027cfedefe35a6a41dff98087"} Sep 10 14:48:03.849157 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.849139 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:48:03.849562 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.849539 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"00b30e5204a38a070be53a9fc0c192535e14d791a18a316a904e5fc5ba748812"} Sep 10 14:48:03.849645 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.849629 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"c90016b66cd03677e2eb3f3e3327d49f07ce8140eca34fe42b265f72a6593434"} Sep 10 14:48:03.849717 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.849677 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"11d0c382bfc3f218c5a6e214712fb5cc02bdd9f184f4e1bd2723f742cbedd562"} Sep 10 14:48:03.849764 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.849723 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"14e32f7d04f4639c93633dc1484f9ffa8fc897ae8e21d2f813aabaf59c5dd736"} Sep 10 14:48:03.851357 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.851332 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-26xrt" event={"ID":"8bfd968c-b1e5-4296-92b7-02f1e672fb2f","Type":"ContainerStarted","Data":"4ee3f8d65a063904edae3f0239abdbc9dcdc210229a36a62241834374e854a92"} Sep 10 14:48:03.878684 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.878641 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-26xrt" podStartSLOduration=4.26355316 podStartE2EDuration="21.878627984s" totalImagesPullingTime="17.615074824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:03.878118336 +0000 UTC m=+22.692701086" watchObservedRunningTime="2026-09-10 14:48:03.878627984 +0000 UTC m=+22.693210785" Sep 10 14:48:03.942316 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:03.942292 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 14:48:04.631064 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:04.630952 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T14:48:03.942313298Z","UUID":"99261c40-6024-4b7b-9cf5-90d7e763a497","Handler":null,"Name":"","Endpoint":""} Sep 10 14:48:04.632901 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:04.632879 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 14:48:04.633019 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:04.632917 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 14:48:04.857651 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:04.857590 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" event={"ID":"fc7a5c70-b722-4d87-8fa0-7caa39ffa412","Type":"ContainerStarted","Data":"73bb00a5f0de0cf50ff023d6fc351a30631dd411a2ded3f8272baa652f04e88a"} Sep 10 14:48:05.658628 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:05.658599 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:05.658628 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:05.658617 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:05.658858 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:05.658736 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:05.658858 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:05.658726 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:48:05.658974 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:05.658873 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:05.658974 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:05.658959 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:48:05.864239 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:05.864165 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:48:05.864726 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:05.864691 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"504eac5598e0e39f18b1adb335ee629ef9ee6226c3b6f694a1e2f09424246296"} Sep 10 14:48:05.867024 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:05.867001 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" event={"ID":"fc7a5c70-b722-4d87-8fa0-7caa39ffa412","Type":"ContainerStarted","Data":"70f60432f593cca08738aee3857193226aa77b78aa6b1d2ac7e227e6cdb9766f"} Sep 10 14:48:05.888835 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:05.887941 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l65cl" podStartSLOduration=5.941167398 podStartE2EDuration="23.887916278s" totalImagesPullingTime="17.94674888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:03.894162384 +0000 UTC m=+22.708745134" watchObservedRunningTime="2026-09-10 14:48:05.887916278 +0000 UTC m=+24.702499032" Sep 10 14:48:07.129850 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:07.129812 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:48:07.130467 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:07.130449 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:48:07.143704 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:07.143652 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlbr4" podStartSLOduration=4.894913074 podStartE2EDuration="25.143630807s" totalImagesPullingTime="20.248717733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:05.888958387 +0000 UTC m=+24.703541142" watchObservedRunningTime="2026-09-10 14:48:07.143630807 +0000 UTC m=+25.958213553" Sep 10 14:48:07.658918 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:07.658668 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:07.659114 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:07.658670 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:07.659114 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:07.659029 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:48:07.659249 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:07.659113 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:07.659249 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:07.658723 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:07.659249 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:07.659219 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:48:09.658474 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.658444 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:09.658474 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.658456 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:09.659200 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.658456 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:09.659200 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:09.658563 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:48:09.659200 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:09.658625 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:09.659200 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:09.658686 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:48:09.878395 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.878371 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:48:09.878756 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.878733 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"2fd42037c4d8bd26622cf0fcef4ff5f72fb5b83af7b7b93975caf19e9bf233af"} Sep 10 14:48:09.879030 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.879010 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:48:09.879203 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.879177 2560 scope.go:117] "RemoveContainer" containerID="3c0a488a384c2a07732a79f25788adf77a62ea0d203cc6ce4f97d801dfb2b127" Sep 10 14:48:09.880948 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.880922 2560 generic.go:356] "Generic (PLEG): container finished" podID="744583dc-6c66-45ed-9532-cd6950609908" containerID="7725926a01fcd511383884bce827e9135ef88fd4c289afcc59472e2695a0491b" exitCode=0 Sep 10 14:48:09.881033 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.880964 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerDied","Data":"7725926a01fcd511383884bce827e9135ef88fd4c289afcc59472e2695a0491b"} Sep 10 14:48:09.895778 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:09.895758 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:48:10.775003 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.774774 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d2txf"] Sep 10 14:48:10.775569 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.775294 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:10.775569 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:10.775436 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:48:10.775706 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.775660 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ts66m"] Sep 10 14:48:10.776419 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.775778 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:10.776419 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:10.775955 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:10.776534 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.776461 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-msnqz"] Sep 10 14:48:10.776589 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.776550 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:10.776646 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:10.776632 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:48:10.886888 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.886863 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:48:10.887211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.887192 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" event={"ID":"9ccb0608-8ad0-48d7-bb71-3f258e830b99","Type":"ContainerStarted","Data":"b8f333b5946d53ba5175e47954211001044a7116de1659afa211895690984c63"} Sep 10 14:48:10.887432 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.887420 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:48:10.887486 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.887462 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:48:10.902777 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.902754 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:48:10.948881 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:10.948736 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" podStartSLOduration=12.296269627 podStartE2EDuration="29.948722916s" totalImagesPullingTime="17.652453289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:10.924836741 +0000 UTC m=+29.739419493" watchObservedRunningTime="2026-09-10 14:48:10.948722916 +0000 UTC m=+29.763305661" Sep 10 14:48:11.695461 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:11.695397 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:48:11.695601 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:11.695499 2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 14:48:11.695956 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:11.695942 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p87s2" Sep 10 14:48:11.891426 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:11.891399 2560 generic.go:356] "Generic (PLEG): container finished" podID="744583dc-6c66-45ed-9532-cd6950609908" containerID="44d61ecc4eec9943c4f06178c2cb7e6a297d1c2007299cb3b66320a9a03c045a" exitCode=0 Sep 10 14:48:11.891903 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:11.891469 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerDied","Data":"44d61ecc4eec9943c4f06178c2cb7e6a297d1c2007299cb3b66320a9a03c045a"} Sep 10 14:48:12.658606 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:12.658566 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:12.658778 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:12.658668 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:12.658778 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:12.658681 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:48:12.658778 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:12.658706 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:12.658967 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:12.658828 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:12.658967 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:12.658902 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:48:13.897256 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:13.897089 2560 generic.go:356] "Generic (PLEG): container finished" podID="744583dc-6c66-45ed-9532-cd6950609908" containerID="20e1244a5903f7624c554688a08f557bd3d8354ee122ad9dd6a6093f6a16004b" exitCode=0 Sep 10 14:48:13.897638 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:13.897162 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerDied","Data":"20e1244a5903f7624c554688a08f557bd3d8354ee122ad9dd6a6093f6a16004b"} Sep 10 14:48:14.659133 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:14.659100 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:14.659310 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:14.659202 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:14.659310 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:14.659235 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:14.659310 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:14.659245 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msnqz" podUID="f4d637a4-3174-4b3b-b117-1e531cba9054" Sep 10 14:48:14.659469 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:14.659319 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2txf" podUID="549f4770-f5a2-49d8-a3ce-d62df642b6e7" Sep 10 14:48:14.659469 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:14.659425 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts66m" podUID="c30f901a-49df-4c7f-a362-4b0ce6f76057" Sep 10 14:48:15.427002 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.426930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:15.427002 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.426970 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:15.427350 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.427067 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:48:15.427350 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.427128 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret podName:549f4770-f5a2-49d8-a3ce-d62df642b6e7 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:47.427114011 +0000 UTC m=+66.241696745 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret") pod "global-pull-secret-syncer-d2txf" (UID: "549f4770-f5a2-49d8-a3ce-d62df642b6e7") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:48:15.427350 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.427073 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:48:15.427350 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.427209 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs podName:c30f901a-49df-4c7f-a362-4b0ce6f76057 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:47.427197559 +0000 UTC m=+66.241780288 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs") pod "network-metrics-daemon-ts66m" (UID: "c30f901a-49df-4c7f-a362-4b0ce6f76057") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:48:15.515988 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.515958 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-4.ec2.internal" event="NodeReady" Sep 10 14:48:15.516171 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.516154 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 14:48:15.527332 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.527310 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:15.527461 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.527447 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:48:15.527518 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.527470 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:48:15.527518 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.527481 2560 projected.go:196] Error preparing data for projected volume kube-api-access-v6vj9 for pod openshift-network-diagnostics/network-check-target-msnqz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:48:15.527611 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.527526 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9 podName:f4d637a4-3174-4b3b-b117-1e531cba9054 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:47.527511258 +0000 UTC m=+66.342093988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6vj9" (UniqueName: "kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9") pod "network-check-target-msnqz" (UID: "f4d637a4-3174-4b3b-b117-1e531cba9054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:48:15.565166 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.565142 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5"] Sep 10 14:48:15.587593 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.587575 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.589888 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.589866 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5"] Sep 10 14:48:15.591096 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.591064 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 14:48:15.591196 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.591136 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 14:48:15.591196 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.591166 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 14:48:15.591306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.591140 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgzsd\"" Sep 10 14:48:15.597146 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.597126 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 14:48:15.599601 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.599581 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4n624"] Sep 10 14:48:15.620468 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.619832 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8d5st"] Sep 10 14:48:15.620648 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.620626 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.623393 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.623370 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 14:48:15.623493 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.623444 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 14:48:15.623622 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.623604 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7c4td\"" Sep 10 14:48:15.634413 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.634394 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8d5st"] Sep 10 14:48:15.634506 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.634429 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4n624"] Sep 10 14:48:15.634566 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.634530 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8d5st" Sep 10 14:48:15.637209 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.637184 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 14:48:15.638186 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.638163 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 14:48:15.638287 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.638186 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q5d77\"" Sep 10 14:48:15.638508 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.638488 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 14:48:15.728609 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728535 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4629d531-0365-416d-8177-92814deff28f-config-volume\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.728609 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728571 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-installation-pull-secrets\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.728609 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ctn\" (UniqueName: \"kubernetes.io/projected/4629d531-0365-416d-8177-92814deff28f-kube-api-access-h9ctn\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.728912 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728658 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-bound-sa-token\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.728912 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728725 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4629d531-0365-416d-8177-92814deff28f-tmp-dir\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.728912 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728769 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-image-registry-private-configuration\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.728912 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728794 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4a99041-6bb6-472c-82d2-184121ef0adf-cert\") pod \"ingress-canary-8d5st\" (UID: \"b4a99041-6bb6-472c-82d2-184121ef0adf\") " pod="openshift-ingress-canary/ingress-canary-8d5st" Sep 10 14:48:15.728912 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728841 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28996ee4-e607-4806-a6f7-243770c7142d-ca-trust-extracted\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.728912 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-registry-certificates\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.728912 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728891 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.729170 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728935 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.729170 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728960 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xf8m\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-kube-api-access-2xf8m\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.729170 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.728982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmz9\" (UniqueName: \"kubernetes.io/projected/b4a99041-6bb6-472c-82d2-184121ef0adf-kube-api-access-4vmz9\") pod \"ingress-canary-8d5st\" (UID: \"b4a99041-6bb6-472c-82d2-184121ef0adf\") " pod="openshift-ingress-canary/ingress-canary-8d5st" Sep 10 14:48:15.729170 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.729011 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-trusted-ca\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.829876 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.829840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.830023 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.829896 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.830023 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.829926 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xf8m\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-kube-api-access-2xf8m\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.830023 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.829953 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmz9\" (UniqueName: \"kubernetes.io/projected/b4a99041-6bb6-472c-82d2-184121ef0adf-kube-api-access-4vmz9\") pod \"ingress-canary-8d5st\" (UID: \"b4a99041-6bb6-472c-82d2-184121ef0adf\") " pod="openshift-ingress-canary/ingress-canary-8d5st" Sep 10 14:48:15.830023 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.829997 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:15.830023 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.830018 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5: secret "image-registry-tls" not found Sep 10 14:48:15.830275 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.830054 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:15.830275 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830079 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-trusted-ca\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.830275 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.830127 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls podName:28996ee4-e607-4806-a6f7-243770c7142d nodeName:}" failed. No retries permitted until 2026-09-10 14:48:16.330105714 +0000 UTC m=+35.144688445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls") pod "image-registry-6ddbb4c9f5-q7mr5" (UID: "28996ee4-e607-4806-a6f7-243770c7142d") : secret "image-registry-tls" not found Sep 10 14:48:15.830275 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:15.830178 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls podName:4629d531-0365-416d-8177-92814deff28f nodeName:}" failed. No retries permitted until 2026-09-10 14:48:16.330146116 +0000 UTC m=+35.144728859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls") pod "dns-default-4n624" (UID: "4629d531-0365-416d-8177-92814deff28f") : secret "dns-default-metrics-tls" not found Sep 10 14:48:15.830275 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830235 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4629d531-0365-416d-8177-92814deff28f-config-volume\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.830275 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830267 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-installation-pull-secrets\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.830604 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830303 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ctn\" (UniqueName: \"kubernetes.io/projected/4629d531-0365-416d-8177-92814deff28f-kube-api-access-h9ctn\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.830604 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830326 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-bound-sa-token\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.830604 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830373 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4629d531-0365-416d-8177-92814deff28f-tmp-dir\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.830604 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830420 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-image-registry-private-configuration\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.830604 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4a99041-6bb6-472c-82d2-184121ef0adf-cert\") pod \"ingress-canary-8d5st\" (UID: \"b4a99041-6bb6-472c-82d2-184121ef0adf\") " pod="openshift-ingress-canary/ingress-canary-8d5st" Sep 10 14:48:15.830604 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830471 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28996ee4-e607-4806-a6f7-243770c7142d-ca-trust-extracted\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.830604 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830498 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-registry-certificates\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.830956 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.830885 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4629d531-0365-416d-8177-92814deff28f-config-volume\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.831067 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.831020 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-registry-certificates\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.831162 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.831072 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28996ee4-e607-4806-a6f7-243770c7142d-ca-trust-extracted\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.831264 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.831249 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-trusted-ca\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.831340 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.831279 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4629d531-0365-416d-8177-92814deff28f-tmp-dir\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.834786 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.834760 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-installation-pull-secrets\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.834914 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.834812 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4a99041-6bb6-472c-82d2-184121ef0adf-cert\") pod \"ingress-canary-8d5st\" (UID: \"b4a99041-6bb6-472c-82d2-184121ef0adf\") " pod="openshift-ingress-canary/ingress-canary-8d5st" Sep 10 14:48:15.834914 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.834790 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-image-registry-private-configuration\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.839738 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.839697 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmz9\" (UniqueName: \"kubernetes.io/projected/b4a99041-6bb6-472c-82d2-184121ef0adf-kube-api-access-4vmz9\") pod \"ingress-canary-8d5st\" (UID: \"b4a99041-6bb6-472c-82d2-184121ef0adf\") " pod="openshift-ingress-canary/ingress-canary-8d5st" Sep 10 14:48:15.840159 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.840133 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ctn\" (UniqueName: \"kubernetes.io/projected/4629d531-0365-416d-8177-92814deff28f-kube-api-access-h9ctn\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:15.840159 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.840158 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-bound-sa-token\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.840463 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.840445 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xf8m\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-kube-api-access-2xf8m\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:15.946235 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:15.946201 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8d5st" Sep 10 14:48:16.261023 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.260424 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8d5st"] Sep 10 14:48:16.264439 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:16.264402 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a99041_6bb6_472c_82d2_184121ef0adf.slice/crio-55bf280e68d748f4bdcf493bad46cea945513fd39cc3ac636072c6c90bcd61f0 WatchSource:0}: Error finding container 55bf280e68d748f4bdcf493bad46cea945513fd39cc3ac636072c6c90bcd61f0: Status 404 returned error can't find the container with id 55bf280e68d748f4bdcf493bad46cea945513fd39cc3ac636072c6c90bcd61f0 Sep 10 14:48:16.341328 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.341251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:16.341328 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.341287 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:16.341594 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:16.341392 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:16.341594 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:16.341410 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5: secret "image-registry-tls" not found Sep 10 14:48:16.341594 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:16.341444 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:16.341594 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:16.341479 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls podName:28996ee4-e607-4806-a6f7-243770c7142d nodeName:}" failed. No retries permitted until 2026-09-10 14:48:17.34145948 +0000 UTC m=+36.156042211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls") pod "image-registry-6ddbb4c9f5-q7mr5" (UID: "28996ee4-e607-4806-a6f7-243770c7142d") : secret "image-registry-tls" not found Sep 10 14:48:16.341594 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:16.341496 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls podName:4629d531-0365-416d-8177-92814deff28f nodeName:}" failed. No retries permitted until 2026-09-10 14:48:17.341487877 +0000 UTC m=+36.156070605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls") pod "dns-default-4n624" (UID: "4629d531-0365-416d-8177-92814deff28f") : secret "dns-default-metrics-tls" not found Sep 10 14:48:16.659167 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.659135 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:16.660005 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.659140 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:16.660005 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.659140 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:16.661670 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.661654 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 14:48:16.662316 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.662299 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 14:48:16.662553 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.662532 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 14:48:16.662642 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.662610 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fr4wz\"" Sep 10 14:48:16.662642 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.662617 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sc2gb\"" Sep 10 14:48:16.662642 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.662631 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 14:48:16.905119 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:16.905087 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8d5st" event={"ID":"b4a99041-6bb6-472c-82d2-184121ef0adf","Type":"ContainerStarted","Data":"55bf280e68d748f4bdcf493bad46cea945513fd39cc3ac636072c6c90bcd61f0"} Sep 10 14:48:17.350104 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:17.350067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:17.350289 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:17.350197 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:17.350289 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:17.350228 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:17.350289 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:17.350289 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls podName:4629d531-0365-416d-8177-92814deff28f nodeName:}" failed. No retries permitted until 2026-09-10 14:48:19.350271486 +0000 UTC m=+38.164854214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls") pod "dns-default-4n624" (UID: "4629d531-0365-416d-8177-92814deff28f") : secret "dns-default-metrics-tls" not found Sep 10 14:48:17.350462 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:17.350302 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:17.350462 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:17.350315 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5: secret "image-registry-tls" not found Sep 10 14:48:17.350462 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:17.350373 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls podName:28996ee4-e607-4806-a6f7-243770c7142d nodeName:}" failed. No retries permitted until 2026-09-10 14:48:19.350356653 +0000 UTC m=+38.164939389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls") pod "image-registry-6ddbb4c9f5-q7mr5" (UID: "28996ee4-e607-4806-a6f7-243770c7142d") : secret "image-registry-tls" not found Sep 10 14:48:18.827424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:18.827178 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx"] Sep 10 14:48:18.838519 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:18.838459 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" Sep 10 14:48:18.839928 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:18.839867 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx"] Sep 10 14:48:18.841563 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:18.841529 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m7s8z\"" Sep 10 14:48:18.841865 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:18.841840 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 14:48:18.842616 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:18.842594 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 14:48:18.965919 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:18.965888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqr9z\" (UniqueName: \"kubernetes.io/projected/297067da-9436-4bae-9040-dd7911d76094-kube-api-access-zqr9z\") pod \"migrator-fdbb8fdff-c79tx\" (UID: \"297067da-9436-4bae-9040-dd7911d76094\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" Sep 10 14:48:19.066509 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.066472 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqr9z\" (UniqueName: \"kubernetes.io/projected/297067da-9436-4bae-9040-dd7911d76094-kube-api-access-zqr9z\") pod \"migrator-fdbb8fdff-c79tx\" (UID: \"297067da-9436-4bae-9040-dd7911d76094\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" Sep 10 14:48:19.075401 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.075366 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqr9z\" (UniqueName: \"kubernetes.io/projected/297067da-9436-4bae-9040-dd7911d76094-kube-api-access-zqr9z\") pod \"migrator-fdbb8fdff-c79tx\" (UID: \"297067da-9436-4bae-9040-dd7911d76094\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" Sep 10 14:48:19.149216 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.149184 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" Sep 10 14:48:19.368523 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.368487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:19.368523 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.368527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:19.368765 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:19.368658 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:19.368765 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:19.368666 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:19.368765 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:19.368689 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5: secret "image-registry-tls" not found Sep 10 14:48:19.368765 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:19.368726 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls podName:4629d531-0365-416d-8177-92814deff28f nodeName:}" failed. No retries permitted until 2026-09-10 14:48:23.368707586 +0000 UTC m=+42.183290316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls") pod "dns-default-4n624" (UID: "4629d531-0365-416d-8177-92814deff28f") : secret "dns-default-metrics-tls" not found Sep 10 14:48:19.368765 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:19.368744 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls podName:28996ee4-e607-4806-a6f7-243770c7142d nodeName:}" failed. No retries permitted until 2026-09-10 14:48:23.36873549 +0000 UTC m=+42.183318226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls") pod "image-registry-6ddbb4c9f5-q7mr5" (UID: "28996ee4-e607-4806-a6f7-243770c7142d") : secret "image-registry-tls" not found Sep 10 14:48:19.822756 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.822659 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx"] Sep 10 14:48:19.840705 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:19.840675 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297067da_9436_4bae_9040_dd7911d76094.slice/crio-2982b176e9011c1bcfbd758417c77c2271a91da4610ad158bf8cde4faae8423e WatchSource:0}: Error finding container 2982b176e9011c1bcfbd758417c77c2271a91da4610ad158bf8cde4faae8423e: Status 404 returned error can't find the container with id 2982b176e9011c1bcfbd758417c77c2271a91da4610ad158bf8cde4faae8423e Sep 10 14:48:19.913527 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.913501 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8d5st" event={"ID":"b4a99041-6bb6-472c-82d2-184121ef0adf","Type":"ContainerStarted","Data":"b025b748d51f4db1fb2bed59819677674271dbb388832472bee365a891f0e311"} Sep 10 14:48:19.914665 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.914642 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" event={"ID":"297067da-9436-4bae-9040-dd7911d76094","Type":"ContainerStarted","Data":"2982b176e9011c1bcfbd758417c77c2271a91da4610ad158bf8cde4faae8423e"} Sep 10 14:48:19.917399 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.917380 2560 generic.go:356] "Generic (PLEG): container finished" podID="744583dc-6c66-45ed-9532-cd6950609908" containerID="86bd91d117c24e2ff8f51f7b40d06d67943ffadd86828b4c06802924422b087e" exitCode=0 Sep 10 14:48:19.917475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.917408 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerDied","Data":"86bd91d117c24e2ff8f51f7b40d06d67943ffadd86828b4c06802924422b087e"} Sep 10 14:48:19.929559 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:19.929524 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8d5st" podStartSLOduration=1.63059771 podStartE2EDuration="4.929512032s" totalImagesPullingTime="3.298914322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:19.929001723 +0000 UTC m=+38.743584474" watchObservedRunningTime="2026-09-10 14:48:19.929512032 +0000 UTC m=+38.744094783" Sep 10 14:48:20.475277 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.475096 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-2kklr"] Sep 10 14:48:20.494614 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.494595 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-2kklr"] Sep 10 14:48:20.494734 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.494679 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.497179 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.497116 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 14:48:20.497179 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.497118 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 14:48:20.497179 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.497194 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 14:48:20.497452 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.497215 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 14:48:20.497452 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.497198 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fwj4k\"" Sep 10 14:48:20.678587 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.678558 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv2v5\" (UniqueName: \"kubernetes.io/projected/80182c73-0969-42d1-8069-536e7e083ab9-kube-api-access-qv2v5\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.678745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.678609 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/80182c73-0969-42d1-8069-536e7e083ab9-signing-key\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.678745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.678641 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/80182c73-0969-42d1-8069-536e7e083ab9-signing-cabundle\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.780023 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.779992 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/80182c73-0969-42d1-8069-536e7e083ab9-signing-key\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.780185 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.780035 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/80182c73-0969-42d1-8069-536e7e083ab9-signing-cabundle\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.780328 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.780278 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2v5\" (UniqueName: \"kubernetes.io/projected/80182c73-0969-42d1-8069-536e7e083ab9-kube-api-access-qv2v5\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.780784 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.780761 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/80182c73-0969-42d1-8069-536e7e083ab9-signing-cabundle\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.784224 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.784195 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/80182c73-0969-42d1-8069-536e7e083ab9-signing-key\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.789141 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.789089 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2v5\" (UniqueName: \"kubernetes.io/projected/80182c73-0969-42d1-8069-536e7e083ab9-kube-api-access-qv2v5\") pod \"service-ca-75d99c484-2kklr\" (UID: \"80182c73-0969-42d1-8069-536e7e083ab9\") " pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.802992 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.802967 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-2kklr" Sep 10 14:48:20.925426 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.925392 2560 generic.go:356] "Generic (PLEG): container finished" podID="744583dc-6c66-45ed-9532-cd6950609908" containerID="010e3ba7a20419222f0003154d8b3a6bcc7ced1fe0494584ff4b0c92ebe8b322" exitCode=0 Sep 10 14:48:20.925770 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:20.925466 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerDied","Data":"010e3ba7a20419222f0003154d8b3a6bcc7ced1fe0494584ff4b0c92ebe8b322"} Sep 10 14:48:21.080514 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:21.080481 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-2kklr"] Sep 10 14:48:21.286657 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:21.286573 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80182c73_0969_42d1_8069_536e7e083ab9.slice/crio-eef3c150e31d06146a11f736e3e3c3d32e6cfa582c071874ac7afbabb1cfa45d WatchSource:0}: Error finding container eef3c150e31d06146a11f736e3e3c3d32e6cfa582c071874ac7afbabb1cfa45d: Status 404 returned error can't find the container with id eef3c150e31d06146a11f736e3e3c3d32e6cfa582c071874ac7afbabb1cfa45d Sep 10 14:48:21.932541 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:21.932512 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56k7w" event={"ID":"744583dc-6c66-45ed-9532-cd6950609908","Type":"ContainerStarted","Data":"01b6564b18917939ccdd40e4dab6c2ccd8b89a4e37ee9d3ce31b59ac6311717c"} Sep 10 14:48:21.934575 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:21.934552 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" event={"ID":"297067da-9436-4bae-9040-dd7911d76094","Type":"ContainerStarted","Data":"b7d3292776237c4c19acef7f6ecf6409f5acc328dd81099596411c77b9f209e9"} Sep 10 14:48:21.934760 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:21.934597 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" event={"ID":"297067da-9436-4bae-9040-dd7911d76094","Type":"ContainerStarted","Data":"66933ae94635a81cc5d6b591a0a063b9a435149741a4bc34910fa295d8366f97"} Sep 10 14:48:21.935855 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:21.935828 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-2kklr" event={"ID":"80182c73-0969-42d1-8069-536e7e083ab9","Type":"ContainerStarted","Data":"eef3c150e31d06146a11f736e3e3c3d32e6cfa582c071874ac7afbabb1cfa45d"} Sep 10 14:48:21.957708 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:21.957667 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-56k7w" podStartSLOduration=9.897682798 podStartE2EDuration="40.957653357s" totalImagesPullingTime="31.059970559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:21.956579983 +0000 UTC m=+40.771162734" watchObservedRunningTime="2026-09-10 14:48:21.957653357 +0000 UTC m=+40.772236107" Sep 10 14:48:21.972575 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:21.972531 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-c79tx" podStartSLOduration=2.22339004 podStartE2EDuration="3.972515498s" totalImagesPullingTime="1.749125458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:21.972159845 +0000 UTC m=+40.786742596" watchObservedRunningTime="2026-09-10 14:48:21.972515498 +0000 UTC m=+40.787098254" Sep 10 14:48:23.404108 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:23.404067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:23.404569 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:23.404130 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:23.404569 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:23.404292 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 14:48:23.404569 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:23.404317 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5: secret "image-registry-tls" not found Sep 10 14:48:23.404569 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:23.404328 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:48:23.404569 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:23.404394 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls podName:28996ee4-e607-4806-a6f7-243770c7142d nodeName:}" failed. No retries permitted until 2026-09-10 14:48:31.404371212 +0000 UTC m=+50.218953969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls") pod "image-registry-6ddbb4c9f5-q7mr5" (UID: "28996ee4-e607-4806-a6f7-243770c7142d") : secret "image-registry-tls" not found Sep 10 14:48:23.404569 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:23.404416 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls podName:4629d531-0365-416d-8177-92814deff28f nodeName:}" failed. No retries permitted until 2026-09-10 14:48:31.404407723 +0000 UTC m=+50.218990453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls") pod "dns-default-4n624" (UID: "4629d531-0365-416d-8177-92814deff28f") : secret "dns-default-metrics-tls" not found Sep 10 14:48:23.944736 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:23.944708 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-2kklr" event={"ID":"80182c73-0969-42d1-8069-536e7e083ab9","Type":"ContainerStarted","Data":"6a9e3387d841db71b545f048391818554d553e7ad884b5db08b22faff2c678db"} Sep 10 14:48:23.961084 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:23.961026 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-2kklr" podStartSLOduration=1.437875003 podStartE2EDuration="3.961006637s" totalImagesPullingTime="2.523131634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:23.959537632 +0000 UTC m=+42.774120384" watchObservedRunningTime="2026-09-10 14:48:23.961006637 +0000 UTC m=+42.775589389" Sep 10 14:48:31.463062 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.463023 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:31.463497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.463076 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:31.465258 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.465232 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4629d531-0365-416d-8177-92814deff28f-metrics-tls\") pod \"dns-default-4n624\" (UID: \"4629d531-0365-416d-8177-92814deff28f\") " pod="openshift-dns/dns-default-4n624" Sep 10 14:48:31.465379 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.465308 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"image-registry-6ddbb4c9f5-q7mr5\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:31.500347 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.500315 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:31.531560 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.531537 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4n624" Sep 10 14:48:31.771841 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.771735 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5"] Sep 10 14:48:31.775722 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:31.775699 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28996ee4_e607_4806_a6f7_243770c7142d.slice/crio-92133c02c3ea857d7ab43ecd4207bb2613bc8f7b9cc14954b1a6ad6a51f4eb65 WatchSource:0}: Error finding container 92133c02c3ea857d7ab43ecd4207bb2613bc8f7b9cc14954b1a6ad6a51f4eb65: Status 404 returned error can't find the container with id 92133c02c3ea857d7ab43ecd4207bb2613bc8f7b9cc14954b1a6ad6a51f4eb65 Sep 10 14:48:31.803424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.803404 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4n624"] Sep 10 14:48:31.806038 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:31.806008 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4629d531_0365_416d_8177_92814deff28f.slice/crio-619a860aaa2f504534a3649e17c15e061fc88d41a8e95237827ac9d0ca23ccdd WatchSource:0}: Error finding container 619a860aaa2f504534a3649e17c15e061fc88d41a8e95237827ac9d0ca23ccdd: Status 404 returned error can't find the container with id 619a860aaa2f504534a3649e17c15e061fc88d41a8e95237827ac9d0ca23ccdd Sep 10 14:48:31.968615 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.968580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4n624" event={"ID":"4629d531-0365-416d-8177-92814deff28f","Type":"ContainerStarted","Data":"619a860aaa2f504534a3649e17c15e061fc88d41a8e95237827ac9d0ca23ccdd"} Sep 10 14:48:31.970233 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.970201 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" event={"ID":"28996ee4-e607-4806-a6f7-243770c7142d","Type":"ContainerStarted","Data":"9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8"} Sep 10 14:48:31.970373 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.970254 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" event={"ID":"28996ee4-e607-4806-a6f7-243770c7142d","Type":"ContainerStarted","Data":"92133c02c3ea857d7ab43ecd4207bb2613bc8f7b9cc14954b1a6ad6a51f4eb65"} Sep 10 14:48:31.970373 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.970318 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:31.988555 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:31.988511 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" podStartSLOduration=44.98849607 podStartE2EDuration="44.98849607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:31.988010745 +0000 UTC m=+50.802593498" watchObservedRunningTime="2026-09-10 14:48:31.98849607 +0000 UTC m=+50.803078820" Sep 10 14:48:33.979487 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:33.979454 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4n624" event={"ID":"4629d531-0365-416d-8177-92814deff28f","Type":"ContainerStarted","Data":"c14165ded125c75ccd0f4fa705b7ac92e6f24f4438846e9393866942a4a9d3d4"} Sep 10 14:48:33.979952 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:33.979514 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4n624" event={"ID":"4629d531-0365-416d-8177-92814deff28f","Type":"ContainerStarted","Data":"3952f14f62d21da2c25e193f9d20501f33fbce833f94d7a106b936a8bb95ab62"} Sep 10 14:48:33.979952 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:33.979579 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4n624" Sep 10 14:48:33.996085 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:33.996035 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4n624" podStartSLOduration=17.545357598 podStartE2EDuration="18.996022971s" totalImagesPullingTime="1.450665373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:33.994787024 +0000 UTC m=+52.809369775" watchObservedRunningTime="2026-09-10 14:48:33.996022971 +0000 UTC m=+52.810605722" Sep 10 14:48:42.903188 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:42.903155 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92lh7" Sep 10 14:48:43.985281 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:43.985243 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4n624" Sep 10 14:48:44.233576 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.233543 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff"] Sep 10 14:48:44.236823 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.236765 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" Sep 10 14:48:44.239086 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.239063 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rd4lc\"" Sep 10 14:48:44.239208 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.239152 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 14:48:44.248544 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.248520 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff"] Sep 10 14:48:44.341601 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.341571 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bcfb2"] Sep 10 14:48:44.343833 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.343812 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.346732 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.346712 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 14:48:44.346932 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.346911 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 14:48:44.347001 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.346960 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 14:48:44.347001 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.346993 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 14:48:44.352277 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.352261 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g76rw\"" Sep 10 14:48:44.359270 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.359251 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/087e86a0-662f-4251-a2d4-33d14e401415-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-jmzff\" (UID: \"087e86a0-662f-4251-a2d4-33d14e401415\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" Sep 10 14:48:44.361421 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.361399 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bcfb2"] Sep 10 14:48:44.460261 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.460237 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/087e86a0-662f-4251-a2d4-33d14e401415-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-jmzff\" (UID: \"087e86a0-662f-4251-a2d4-33d14e401415\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" Sep 10 14:48:44.460409 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.460269 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b59b8e13-91dd-4cfd-9a36-52b88b999078-crio-socket\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.460409 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.460288 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b59b8e13-91dd-4cfd-9a36-52b88b999078-data-volume\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.460409 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.460316 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzq5\" (UniqueName: \"kubernetes.io/projected/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-api-access-vlzq5\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.460522 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.460403 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.460522 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.460465 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.462582 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.462555 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/087e86a0-662f-4251-a2d4-33d14e401415-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-jmzff\" (UID: \"087e86a0-662f-4251-a2d4-33d14e401415\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" Sep 10 14:48:44.546822 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.546750 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" Sep 10 14:48:44.560837 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.560817 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.560947 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.560900 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b59b8e13-91dd-4cfd-9a36-52b88b999078-crio-socket\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.560947 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:44.560924 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:48:44.561063 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:44.560999 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls podName:b59b8e13-91dd-4cfd-9a36-52b88b999078 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:45.060979437 +0000 UTC m=+63.875562187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcfb2" (UID: "b59b8e13-91dd-4cfd-9a36-52b88b999078") : secret "insights-runtime-extractor-tls" not found Sep 10 14:48:44.561063 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.561018 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b59b8e13-91dd-4cfd-9a36-52b88b999078-crio-socket\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.561063 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.560929 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b59b8e13-91dd-4cfd-9a36-52b88b999078-data-volume\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.561227 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.561093 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzq5\" (UniqueName: \"kubernetes.io/projected/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-api-access-vlzq5\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.561227 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.561137 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.561227 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:44.561203 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:48:44.561227 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.561211 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b59b8e13-91dd-4cfd-9a36-52b88b999078-data-volume\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.561370 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:44.561267 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm podName:b59b8e13-91dd-4cfd-9a36-52b88b999078 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:45.061254025 +0000 UTC m=+63.875836753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bcfb2" (UID: "b59b8e13-91dd-4cfd-9a36-52b88b999078") : configmap "kube-rbac-proxy" not found Sep 10 14:48:44.571730 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.571711 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzq5\" (UniqueName: \"kubernetes.io/projected/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-api-access-vlzq5\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:44.811086 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.811019 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff"] Sep 10 14:48:44.814068 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:44.814045 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087e86a0_662f_4251_a2d4_33d14e401415.slice/crio-46c2d98ff43222d16d387ffac62d3dd78174f30330855ba9a3fd8503895c344e WatchSource:0}: Error finding container 46c2d98ff43222d16d387ffac62d3dd78174f30330855ba9a3fd8503895c344e: Status 404 returned error can't find the container with id 46c2d98ff43222d16d387ffac62d3dd78174f30330855ba9a3fd8503895c344e Sep 10 14:48:44.931078 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:44.931043 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4n624_4629d531-0365-416d-8177-92814deff28f/dns/0.log" Sep 10 14:48:45.016616 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.016587 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" event={"ID":"087e86a0-662f-4251-a2d4-33d14e401415","Type":"ContainerStarted","Data":"46c2d98ff43222d16d387ffac62d3dd78174f30330855ba9a3fd8503895c344e"} Sep 10 14:48:45.064652 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.064581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:45.064652 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.064623 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:45.064860 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:45.064699 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:48:45.064860 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:45.064762 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm podName:b59b8e13-91dd-4cfd-9a36-52b88b999078 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:46.064745617 +0000 UTC m=+64.879328350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bcfb2" (UID: "b59b8e13-91dd-4cfd-9a36-52b88b999078") : configmap "kube-rbac-proxy" not found Sep 10 14:48:45.064860 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:45.064790 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:48:45.064981 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:45.064863 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls podName:b59b8e13-91dd-4cfd-9a36-52b88b999078 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:46.064848011 +0000 UTC m=+64.879430753 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcfb2" (UID: "b59b8e13-91dd-4cfd-9a36-52b88b999078") : secret "insights-runtime-extractor-tls" not found Sep 10 14:48:45.113332 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.113306 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4n624_4629d531-0365-416d-8177-92814deff28f/kube-rbac-proxy/0.log" Sep 10 14:48:45.975740 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.975707 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7695b79b66-t8fvs"] Sep 10 14:48:45.978191 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.978176 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:45.980313 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.980287 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 14:48:45.980454 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.980347 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 14:48:45.981353 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.981283 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 14:48:45.981353 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.981304 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 14:48:45.981353 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.981320 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 14:48:45.981610 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.981592 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h7btv\"" Sep 10 14:48:45.981673 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.981639 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 14:48:45.981673 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.981661 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 14:48:45.989362 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:45.989338 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7695b79b66-t8fvs"] Sep 10 14:48:46.070947 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.070920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fjj2\" (UniqueName: \"kubernetes.io/projected/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-kube-api-access-6fjj2\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.070965 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.070993 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-service-ca\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.071057 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-config\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:46.071080 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.071103 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-serving-cert\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:46.071146 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm podName:b59b8e13-91dd-4cfd-9a36-52b88b999078 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:48.071128313 +0000 UTC m=+66.885711059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bcfb2" (UID: "b59b8e13-91dd-4cfd-9a36-52b88b999078") : configmap "kube-rbac-proxy" not found Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.071178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-oauth-serving-cert\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.071208 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.071275 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-oauth-config\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.071314 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:46.071298 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:48:46.071616 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:46.071334 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls podName:b59b8e13-91dd-4cfd-9a36-52b88b999078 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:48.071322048 +0000 UTC m=+66.885904798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcfb2" (UID: "b59b8e13-91dd-4cfd-9a36-52b88b999078") : secret "insights-runtime-extractor-tls" not found Sep 10 14:48:46.115716 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.115682 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4r7d6_20b78c3e-ce14-498b-bdbf-8baa9718867f/dns-node-resolver/0.log" Sep 10 14:48:46.172305 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.172272 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-oauth-config\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.172443 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.172322 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjj2\" (UniqueName: \"kubernetes.io/projected/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-kube-api-access-6fjj2\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.172443 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.172373 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-service-ca\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.172443 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.172410 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-config\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.172443 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.172436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-serving-cert\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.172594 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.172463 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-oauth-serving-cert\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.173226 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.173199 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-oauth-serving-cert\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.173465 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.173433 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-config\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.173588 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.173564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-service-ca\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.175375 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.175354 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-oauth-config\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.175582 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.175560 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-serving-cert\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.180420 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.180401 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjj2\" (UniqueName: \"kubernetes.io/projected/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-kube-api-access-6fjj2\") pod \"console-7695b79b66-t8fvs\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.287442 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.287361 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:46.560456 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.560381 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7695b79b66-t8fvs"] Sep 10 14:48:46.639063 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:46.639034 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc50ae6e_d0a4_4765_b8ff_b921772e9cd2.slice/crio-5789dacb54b04434f1922f3416fc96f47e1a74f8666fd1430c86c2f84c6ef1ed WatchSource:0}: Error finding container 5789dacb54b04434f1922f3416fc96f47e1a74f8666fd1430c86c2f84c6ef1ed: Status 404 returned error can't find the container with id 5789dacb54b04434f1922f3416fc96f47e1a74f8666fd1430c86c2f84c6ef1ed Sep 10 14:48:46.914181 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:46.914153 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6ddbb4c9f5-q7mr5_28996ee4-e607-4806-a6f7-243770c7142d/registry/0.log" Sep 10 14:48:47.024908 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.024883 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695b79b66-t8fvs" event={"ID":"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2","Type":"ContainerStarted","Data":"5789dacb54b04434f1922f3416fc96f47e1a74f8666fd1430c86c2f84c6ef1ed"} Sep 10 14:48:47.026371 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.026345 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" event={"ID":"087e86a0-662f-4251-a2d4-33d14e401415","Type":"ContainerStarted","Data":"59220d64dc0a93cded442f26312af52cb892766847dbb6d30978c63f94220f47"} Sep 10 14:48:47.026641 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.026618 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" Sep 10 14:48:47.031356 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.031337 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" Sep 10 14:48:47.041513 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.041469 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jmzff" podStartSLOduration=1.189612377 podStartE2EDuration="3.041456701s" totalImagesPullingTime="1.851844324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:47.04132251 +0000 UTC m=+65.855905260" watchObservedRunningTime="2026-09-10 14:48:47.041456701 +0000 UTC m=+65.856039452" Sep 10 14:48:47.483226 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.483165 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:47.483692 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.483291 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:47.486079 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.486046 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 14:48:47.486894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.486874 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 14:48:47.496479 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.496452 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/549f4770-f5a2-49d8-a3ce-d62df642b6e7-original-pull-secret\") pod \"global-pull-secret-syncer-d2txf\" (UID: \"549f4770-f5a2-49d8-a3ce-d62df642b6e7\") " pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:47.496624 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.496604 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30f901a-49df-4c7f-a362-4b0ce6f76057-metrics-certs\") pod \"network-metrics-daemon-ts66m\" (UID: \"c30f901a-49df-4c7f-a362-4b0ce6f76057\") " pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:47.574161 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.574128 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2txf" Sep 10 14:48:47.581193 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.581172 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fr4wz\"" Sep 10 14:48:47.584497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.584472 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:47.587052 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.587020 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 14:48:47.590021 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.589998 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts66m" Sep 10 14:48:47.597447 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.597427 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 14:48:47.608286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.608259 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vj9\" (UniqueName: \"kubernetes.io/projected/f4d637a4-3174-4b3b-b117-1e531cba9054-kube-api-access-v6vj9\") pod \"network-check-target-msnqz\" (UID: \"f4d637a4-3174-4b3b-b117-1e531cba9054\") " pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:47.713385 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.713359 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l298v_a8eb91f7-476a-4c3a-b163-adac5bc16180/node-ca/0.log" Sep 10 14:48:47.845721 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.845603 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d2txf"] Sep 10 14:48:47.848810 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:47.848771 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549f4770_f5a2_49d8_a3ce_d62df642b6e7.slice/crio-cfa4a39471143d0735e1ddbf78df3f9a5d0d166da98088b4f90c3254a725bdf1 WatchSource:0}: Error finding container cfa4a39471143d0735e1ddbf78df3f9a5d0d166da98088b4f90c3254a725bdf1: Status 404 returned error can't find the container with id cfa4a39471143d0735e1ddbf78df3f9a5d0d166da98088b4f90c3254a725bdf1 Sep 10 14:48:47.871551 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.871529 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sc2gb\"" Sep 10 14:48:47.871551 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.871543 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ts66m"] Sep 10 14:48:47.874664 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:47.874641 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30f901a_49df_4c7f_a362_4b0ce6f76057.slice/crio-9d4d73517bffe85d49ff6abb864efd15251adde537295f0859fa2fb18ec870ef WatchSource:0}: Error finding container 9d4d73517bffe85d49ff6abb864efd15251adde537295f0859fa2fb18ec870ef: Status 404 returned error can't find the container with id 9d4d73517bffe85d49ff6abb864efd15251adde537295f0859fa2fb18ec870ef Sep 10 14:48:47.880101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:47.880079 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:48.031468 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:48.031429 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d2txf" event={"ID":"549f4770-f5a2-49d8-a3ce-d62df642b6e7","Type":"ContainerStarted","Data":"cfa4a39471143d0735e1ddbf78df3f9a5d0d166da98088b4f90c3254a725bdf1"} Sep 10 14:48:48.032739 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:48.032711 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ts66m" event={"ID":"c30f901a-49df-4c7f-a362-4b0ce6f76057","Type":"ContainerStarted","Data":"9d4d73517bffe85d49ff6abb864efd15251adde537295f0859fa2fb18ec870ef"} Sep 10 14:48:48.088734 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:48.088708 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:48:48.088890 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:48.088779 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm podName:b59b8e13-91dd-4cfd-9a36-52b88b999078 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:52.088761074 +0000 UTC m=+70.903343816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bcfb2" (UID: "b59b8e13-91dd-4cfd-9a36-52b88b999078") : configmap "kube-rbac-proxy" not found Sep 10 14:48:48.088890 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:48.088647 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:48.089009 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:48.088919 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:48.089078 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:48.089023 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:48:48.089078 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:48.089064 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls podName:b59b8e13-91dd-4cfd-9a36-52b88b999078 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:52.089052795 +0000 UTC m=+70.903635524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bcfb2" (UID: "b59b8e13-91dd-4cfd-9a36-52b88b999078") : secret "insights-runtime-extractor-tls" not found Sep 10 14:48:48.112531 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:48.112468 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8d5st_b4a99041-6bb6-472c-82d2-184121ef0adf/serve-healthcheck-canary/0.log" Sep 10 14:48:48.169718 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:48.169689 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-msnqz"] Sep 10 14:48:48.172935 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:48.172906 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d637a4_3174_4b3b_b117_1e531cba9054.slice/crio-99efb8a064e0c1dd05ecf6d94c86522bcefe1f27d55a08b63d8fb6c29ef2024f WatchSource:0}: Error finding container 99efb8a064e0c1dd05ecf6d94c86522bcefe1f27d55a08b63d8fb6c29ef2024f: Status 404 returned error can't find the container with id 99efb8a064e0c1dd05ecf6d94c86522bcefe1f27d55a08b63d8fb6c29ef2024f Sep 10 14:48:49.038936 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:49.038857 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-msnqz" event={"ID":"f4d637a4-3174-4b3b-b117-1e531cba9054","Type":"ContainerStarted","Data":"99efb8a064e0c1dd05ecf6d94c86522bcefe1f27d55a08b63d8fb6c29ef2024f"} Sep 10 14:48:51.050279 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:51.050236 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695b79b66-t8fvs" event={"ID":"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2","Type":"ContainerStarted","Data":"ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b"} Sep 10 14:48:51.052276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:51.052242 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ts66m" event={"ID":"c30f901a-49df-4c7f-a362-4b0ce6f76057","Type":"ContainerStarted","Data":"acf848e9f6870a2c67eaf1fed731157f565f7cdc965e6eac4f05d9e8a3b0bdd2"} Sep 10 14:48:51.052401 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:51.052300 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ts66m" event={"ID":"c30f901a-49df-4c7f-a362-4b0ce6f76057","Type":"ContainerStarted","Data":"f901e0e673ae1e23d6bca1ac477bcc100accf67fc05a18f5e722f8d664292ebd"} Sep 10 14:48:51.066331 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:51.066273 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7695b79b66-t8fvs" podStartSLOduration=2.409104833 podStartE2EDuration="6.066257762s" totalImagesPullingTime="3.657152929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:51.06603742 +0000 UTC m=+69.880620199" watchObservedRunningTime="2026-09-10 14:48:51.066257762 +0000 UTC m=+69.880840513" Sep 10 14:48:51.081321 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:51.081272 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ts66m" podStartSLOduration=66.662319433 podStartE2EDuration="1m9.081253243s" totalImagesPullingTime="2.41893381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:51.079916726 +0000 UTC m=+69.894499478" watchObservedRunningTime="2026-09-10 14:48:51.081253243 +0000 UTC m=+69.895836028" Sep 10 14:48:52.124968 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:52.124926 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:52.125479 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:52.125035 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:52.125606 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:52.125586 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b59b8e13-91dd-4cfd-9a36-52b88b999078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:52.127865 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:52.127838 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b59b8e13-91dd-4cfd-9a36-52b88b999078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bcfb2\" (UID: \"b59b8e13-91dd-4cfd-9a36-52b88b999078\") " pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:52.151902 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:52.151881 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bcfb2" Sep 10 14:48:52.436409 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:52.436324 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bcfb2"] Sep 10 14:48:52.980628 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:52.980599 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:48:53.062980 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.062941 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d2txf" event={"ID":"549f4770-f5a2-49d8-a3ce-d62df642b6e7","Type":"ContainerStarted","Data":"1750f3deb1a362c1d58b2236bada63c02531a931f5fc14856216037f36a17f80"} Sep 10 14:48:53.167499 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:53.167455 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59b8e13_91dd_4cfd_9a36_52b88b999078.slice/crio-c3b48b80133cc6c03fc2134ac53b8f50b178a331e1c6cc3b9356ee4a7c95a010 WatchSource:0}: Error finding container c3b48b80133cc6c03fc2134ac53b8f50b178a331e1c6cc3b9356ee4a7c95a010: Status 404 returned error can't find the container with id c3b48b80133cc6c03fc2134ac53b8f50b178a331e1c6cc3b9356ee4a7c95a010 Sep 10 14:48:53.654546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.654493 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d2txf" podStartSLOduration=67.360767593 podStartE2EDuration="1m11.654479085s" totalImagesPullingTime="4.293711492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:53.077515998 +0000 UTC m=+71.892098750" watchObservedRunningTime="2026-09-10 14:48:53.654479085 +0000 UTC m=+72.469061813" Sep 10 14:48:53.655025 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.655006 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr"] Sep 10 14:48:53.678164 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.678137 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.681119 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.681097 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 14:48:53.681817 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.681573 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 14:48:53.681817 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.681597 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wb742\"" Sep 10 14:48:53.681817 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.681635 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 14:48:53.681817 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.681689 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 14:48:53.682080 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.681931 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 14:48:53.682080 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.681934 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 14:48:53.683682 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.683507 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr"] Sep 10 14:48:53.684190 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.684173 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zfnbw"] Sep 10 14:48:53.704184 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.704166 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.706742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.706723 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 14:48:53.706864 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.706787 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 14:48:53.706864 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.706854 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 14:48:53.706981 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.706808 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c5gfh\"" Sep 10 14:48:53.739749 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.739724 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.739869 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.739766 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.739869 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.739814 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.739952 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.739887 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.739952 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.739940 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhgd\" (UniqueName: \"kubernetes.io/projected/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-api-access-2bhgd\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.740031 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.739979 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.841155 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841128 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.841274 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841175 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.841274 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841204 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-sys\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.841274 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841238 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.841274 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841262 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-root\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.841475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841284 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-wtmp\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.841475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841333 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.841475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841365 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.841475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841400 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-tls\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.841475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841439 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhgd\" (UniqueName: \"kubernetes.io/projected/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-api-access-2bhgd\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.841475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841464 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.841754 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841495 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.841754 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841524 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-textfile\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.841754 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841553 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3db175b6-6536-4b9c-854d-0630040d4d33-metrics-client-ca\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.841754 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.841579 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4dm\" (UniqueName: \"kubernetes.io/projected/3db175b6-6536-4b9c-854d-0630040d4d33-kube-api-access-gv4dm\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.842049 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.842010 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.842049 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.842035 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.842183 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.842108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.844155 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.844128 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.844310 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.844291 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.849079 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.849060 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhgd\" (UniqueName: \"kubernetes.io/projected/19e46b0f-b5bd-44fb-aa8f-ae38f5555e01-kube-api-access-2bhgd\") pod \"kube-state-metrics-74cb57df7-bpcgr\" (UID: \"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:53.942958 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.942896 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-tls\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.942958 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.942935 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943110 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.942964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-textfile\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943110 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.942988 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3db175b6-6536-4b9c-854d-0630040d4d33-metrics-client-ca\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943110 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943003 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4dm\" (UniqueName: \"kubernetes.io/projected/3db175b6-6536-4b9c-854d-0630040d4d33-kube-api-access-gv4dm\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943110 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943060 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943110 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943102 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-sys\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943361 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:53.943110 2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 14:48:53.943361 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943137 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-root\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943361 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943167 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-wtmp\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943361 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:48:53.943224 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-tls podName:3db175b6-6536-4b9c-854d-0630040d4d33 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:54.443206291 +0000 UTC m=+73.257789037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-tls") pod "node-exporter-zfnbw" (UID: "3db175b6-6536-4b9c-854d-0630040d4d33") : secret "node-exporter-tls" not found Sep 10 14:48:53.943560 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943346 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-sys\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943560 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943386 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-root\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943560 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-textfile\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943560 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943516 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-wtmp\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943750 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943675 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3db175b6-6536-4b9c-854d-0630040d4d33-metrics-client-ca\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.943901 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.943881 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.945447 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.945428 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.953445 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.953425 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4dm\" (UniqueName: \"kubernetes.io/projected/3db175b6-6536-4b9c-854d-0630040d4d33-kube-api-access-gv4dm\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:53.988853 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:53.988833 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" Sep 10 14:48:54.071144 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.070997 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-msnqz" event={"ID":"f4d637a4-3174-4b3b-b117-1e531cba9054","Type":"ContainerStarted","Data":"b986c2cec8e96511bba832e9f8e46ef09644e4beeb0ce669d3148b28f51bbb63"} Sep 10 14:48:54.071144 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.071096 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:48:54.074408 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.074372 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bcfb2" event={"ID":"b59b8e13-91dd-4cfd-9a36-52b88b999078","Type":"ContainerStarted","Data":"9e4d73366a918ffeec13008719ed0d134db3cf070963d6a25a4dc9bb86b527fa"} Sep 10 14:48:54.074536 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.074424 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bcfb2" event={"ID":"b59b8e13-91dd-4cfd-9a36-52b88b999078","Type":"ContainerStarted","Data":"c3b48b80133cc6c03fc2134ac53b8f50b178a331e1c6cc3b9356ee4a7c95a010"} Sep 10 14:48:54.085918 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.085878 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-msnqz" podStartSLOduration=67.025968243 podStartE2EDuration="1m12.085863674s" totalImagesPullingTime="5.059895431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:54.084668531 +0000 UTC m=+72.899251283" watchObservedRunningTime="2026-09-10 14:48:54.085863674 +0000 UTC m=+72.900446425" Sep 10 14:48:54.285715 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.285651 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr"] Sep 10 14:48:54.288755 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:54.288728 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e46b0f_b5bd_44fb_aa8f_ae38f5555e01.slice/crio-f910fab7a18741008b6d69666c07569742e74dc4990ee0823ff336e4976d5ee1 WatchSource:0}: Error finding container f910fab7a18741008b6d69666c07569742e74dc4990ee0823ff336e4976d5ee1: Status 404 returned error can't find the container with id f910fab7a18741008b6d69666c07569742e74dc4990ee0823ff336e4976d5ee1 Sep 10 14:48:54.448764 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.448731 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-tls\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:54.450906 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.450880 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3db175b6-6536-4b9c-854d-0630040d4d33-node-exporter-tls\") pod \"node-exporter-zfnbw\" (UID: \"3db175b6-6536-4b9c-854d-0630040d4d33\") " pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:54.613628 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.613563 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zfnbw" Sep 10 14:48:54.716529 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.716496 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:48:54.731329 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.731304 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.733816 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.733780 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:48:54.733936 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.733918 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 14:48:54.733995 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.733923 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 14:48:54.734042 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.733923 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d2kgl\"" Sep 10 14:48:54.734154 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.734138 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 14:48:54.734362 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.734343 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 14:48:54.734459 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.734361 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 14:48:54.734459 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.734378 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 14:48:54.734638 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.734611 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 14:48:54.734749 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.734682 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 14:48:54.734851 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.734747 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 14:48:54.746947 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:54.746922 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db175b6_6536_4b9c_854d_0630040d4d33.slice/crio-e78f392f3806fc9f7584c2d0233da738c4db61b96f7353e5bf9ea0588b5e5490 WatchSource:0}: Error finding container e78f392f3806fc9f7584c2d0233da738c4db61b96f7353e5bf9ea0588b5e5490: Status 404 returned error can't find the container with id e78f392f3806fc9f7584c2d0233da738c4db61b96f7353e5bf9ea0588b5e5490 Sep 10 14:48:54.851760 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.851732 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.851900 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.851772 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.851900 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.851789 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx96z\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-kube-api-access-mx96z\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.851900 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.851855 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.851900 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.851872 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.851900 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.851885 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.851900 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.851901 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.852145 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.851959 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-out\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.852145 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.852008 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.852145 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.852038 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.852145 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.852064 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.852145 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.852093 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-web-config\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.852145 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.852118 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952637 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952614 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952748 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952748 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952748 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952728 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx96z\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-kube-api-access-mx96z\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952919 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952755 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952919 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952781 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952919 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952919 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952846 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952919 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952877 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-out\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.952919 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952904 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.953211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.953211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952968 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.953211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.952997 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-web-config\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.953659 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.953630 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.953824 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.953735 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.955106 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.954573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.955721 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.955695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.955828 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.955766 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.955828 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.955785 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-web-config\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.955936 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.955906 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-out\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.956153 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.956131 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.956289 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.956268 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.956334 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.956270 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.957099 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.957077 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-volume\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.957591 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.957574 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:54.960774 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:54.960756 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx96z\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-kube-api-access-mx96z\") pod \"alertmanager-main-0\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:55.044356 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.044334 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:48:55.079439 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.079409 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bcfb2" event={"ID":"b59b8e13-91dd-4cfd-9a36-52b88b999078","Type":"ContainerStarted","Data":"14e6a6320b9df4d57fbfea6d9fb421695aff1c075343671ebcb29681385bf8ba"} Sep 10 14:48:55.080538 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.080509 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" event={"ID":"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01","Type":"ContainerStarted","Data":"f910fab7a18741008b6d69666c07569742e74dc4990ee0823ff336e4976d5ee1"} Sep 10 14:48:55.081765 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.081743 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfnbw" event={"ID":"3db175b6-6536-4b9c-854d-0630040d4d33","Type":"ContainerStarted","Data":"e78f392f3806fc9f7584c2d0233da738c4db61b96f7353e5bf9ea0588b5e5490"} Sep 10 14:48:55.338389 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.338317 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:48:55.342739 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:55.342709 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d54500a_fbcb_4889_8a5b_5c330efb9c94.slice/crio-3454c2d573f39111adf88043d1cadd29cfe8ee28d7671c68b92fb8dcad62833e WatchSource:0}: Error finding container 3454c2d573f39111adf88043d1cadd29cfe8ee28d7671c68b92fb8dcad62833e: Status 404 returned error can't find the container with id 3454c2d573f39111adf88043d1cadd29cfe8ee28d7671c68b92fb8dcad62833e Sep 10 14:48:55.630374 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.630341 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-764bd5c656-xb7f4"] Sep 10 14:48:55.645064 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.645025 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-764bd5c656-xb7f4"] Sep 10 14:48:55.645198 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.645172 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.648482 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.648437 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 14:48:55.648602 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.648513 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 14:48:55.648683 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.648666 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 14:48:55.648742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.648711 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 14:48:55.649078 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.649026 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 14:48:55.649456 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.649237 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eotdegh06a79p\"" Sep 10 14:48:55.649456 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.649336 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hnwgh\"" Sep 10 14:48:55.758968 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.758929 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.758968 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.758966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.759221 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.759005 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.759221 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.759077 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhnf7\" (UniqueName: \"kubernetes.io/projected/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-kube-api-access-nhnf7\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.759221 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.759163 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.759221 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.759203 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-tls\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.759429 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.759231 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-metrics-client-ca\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.759429 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.759282 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-grpc-tls\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.861098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.860158 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.861098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.860213 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.861098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.860256 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.861098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.860285 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhnf7\" (UniqueName: \"kubernetes.io/projected/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-kube-api-access-nhnf7\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.861098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.860333 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.861098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.860367 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-tls\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.861098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.860392 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-metrics-client-ca\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.861098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.860419 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-grpc-tls\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.862401 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.862369 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-metrics-client-ca\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.864429 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.864397 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-tls\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.865070 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.865040 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.865174 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.865145 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-grpc-tls\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.865174 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.865158 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.867228 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.867187 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.868213 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.868187 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.869836 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.869775 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhnf7\" (UniqueName: \"kubernetes.io/projected/3fa1eb81-b11f-4bc0-8794-9b05406dbc23-kube-api-access-nhnf7\") pod \"thanos-querier-764bd5c656-xb7f4\" (UID: \"3fa1eb81-b11f-4bc0-8794-9b05406dbc23\") " pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:55.958386 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:55.958314 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:48:56.088601 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:56.088563 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerStarted","Data":"3454c2d573f39111adf88043d1cadd29cfe8ee28d7671c68b92fb8dcad62833e"} Sep 10 14:48:56.287794 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:56.287705 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:56.287794 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:56.287784 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:56.293404 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:56.293387 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:57.096996 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:57.096963 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:48:57.502306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:57.502282 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-764bd5c656-xb7f4"] Sep 10 14:48:57.504692 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:57.504670 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa1eb81_b11f_4bc0_8794_9b05406dbc23.slice/crio-476fac579c1324f561cc99d553c6d293352c08b4aab8a15a1021a2afb463ae66 WatchSource:0}: Error finding container 476fac579c1324f561cc99d553c6d293352c08b4aab8a15a1021a2afb463ae66: Status 404 returned error can't find the container with id 476fac579c1324f561cc99d553c6d293352c08b4aab8a15a1021a2afb463ae66 Sep 10 14:48:58.097584 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.097541 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" event={"ID":"3fa1eb81-b11f-4bc0-8794-9b05406dbc23","Type":"ContainerStarted","Data":"476fac579c1324f561cc99d553c6d293352c08b4aab8a15a1021a2afb463ae66"} Sep 10 14:48:58.099593 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.099567 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bcfb2" event={"ID":"b59b8e13-91dd-4cfd-9a36-52b88b999078","Type":"ContainerStarted","Data":"bdcfb10c96060e44a6ccf10fc11558308885eda92cacf25bc609d0ff9ed86689"} Sep 10 14:48:58.101547 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.101519 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" event={"ID":"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01","Type":"ContainerStarted","Data":"e1adee53ab186a100348fa1859c011a51e80e7b0cc6bc5d51bc5e6fa586c9cf8"} Sep 10 14:48:58.101666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.101572 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" event={"ID":"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01","Type":"ContainerStarted","Data":"a7374a5cf9dec65c44709d94f740cb5df45e22a387b8380b5f09e8bb4c24960b"} Sep 10 14:48:58.101666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.101603 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" event={"ID":"19e46b0f-b5bd-44fb-aa8f-ae38f5555e01","Type":"ContainerStarted","Data":"7d49d1d3f7abd6c0485494cbcaf9bc137107b6cfc1fb8acda1f2ea932fab40d7"} Sep 10 14:48:58.103118 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.103094 2560 generic.go:356] "Generic (PLEG): container finished" podID="3db175b6-6536-4b9c-854d-0630040d4d33" containerID="42e5885cf86eae2931c95ecc1bc12a2a773a8c152c32972308c96810945c29f2" exitCode=0 Sep 10 14:48:58.103209 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.103153 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfnbw" event={"ID":"3db175b6-6536-4b9c-854d-0630040d4d33","Type":"ContainerDied","Data":"42e5885cf86eae2931c95ecc1bc12a2a773a8c152c32972308c96810945c29f2"} Sep 10 14:48:58.104849 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.104825 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerID="ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5" exitCode=0 Sep 10 14:48:58.104930 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.104890 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerDied","Data":"ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5"} Sep 10 14:48:58.116382 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.116336 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bcfb2" podStartSLOduration=10.256390256 podStartE2EDuration="14.116320671s" totalImagesPullingTime="3.859930415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:48:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:58.116168265 +0000 UTC m=+76.930751017" watchObservedRunningTime="2026-09-10 14:48:58.116320671 +0000 UTC m=+76.930903422" Sep 10 14:48:58.184052 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.184012 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-bpcgr" podStartSLOduration=2.246652872 podStartE2EDuration="5.183996165s" totalImagesPullingTime="2.937343293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:58.183321519 +0000 UTC m=+76.997904272" watchObservedRunningTime="2026-09-10 14:48:58.183996165 +0000 UTC m=+76.998578916" Sep 10 14:48:58.449041 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.449009 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d"] Sep 10 14:48:58.451334 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.451316 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:48:58.453661 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.453644 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 14:48:58.454220 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.454202 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tqkph\"" Sep 10 14:48:58.460700 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.460677 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d"] Sep 10 14:48:58.586282 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.586247 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41904cb2-4d32-486c-bf85-b6c601e61b12-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-s9n4d\" (UID: \"41904cb2-4d32-486c-bf85-b6c601e61b12\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:48:58.586282 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.586280 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzw8\" (UniqueName: \"kubernetes.io/projected/41904cb2-4d32-486c-bf85-b6c601e61b12-kube-api-access-vbzw8\") pod \"monitoring-plugin-85bbbbbf4-s9n4d\" (UID: \"41904cb2-4d32-486c-bf85-b6c601e61b12\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:48:58.687827 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.687782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41904cb2-4d32-486c-bf85-b6c601e61b12-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-s9n4d\" (UID: \"41904cb2-4d32-486c-bf85-b6c601e61b12\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:48:58.687827 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.687835 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzw8\" (UniqueName: \"kubernetes.io/projected/41904cb2-4d32-486c-bf85-b6c601e61b12-kube-api-access-vbzw8\") pod \"monitoring-plugin-85bbbbbf4-s9n4d\" (UID: \"41904cb2-4d32-486c-bf85-b6c601e61b12\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:48:58.690378 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.690342 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41904cb2-4d32-486c-bf85-b6c601e61b12-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-s9n4d\" (UID: \"41904cb2-4d32-486c-bf85-b6c601e61b12\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:48:58.699586 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.699521 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzw8\" (UniqueName: \"kubernetes.io/projected/41904cb2-4d32-486c-bf85-b6c601e61b12-kube-api-access-vbzw8\") pod \"monitoring-plugin-85bbbbbf4-s9n4d\" (UID: \"41904cb2-4d32-486c-bf85-b6c601e61b12\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:48:58.761503 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:58.761461 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:48:59.040549 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.040397 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d"] Sep 10 14:48:59.044244 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:48:59.044211 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41904cb2_4d32_486c_bf85_b6c601e61b12.slice/crio-bc6a448003d38f954fcdae51202375cd0fe22d518f937b02e641b8f4a6afc854 WatchSource:0}: Error finding container bc6a448003d38f954fcdae51202375cd0fe22d518f937b02e641b8f4a6afc854: Status 404 returned error can't find the container with id bc6a448003d38f954fcdae51202375cd0fe22d518f937b02e641b8f4a6afc854 Sep 10 14:48:59.113575 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.113533 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfnbw" event={"ID":"3db175b6-6536-4b9c-854d-0630040d4d33","Type":"ContainerStarted","Data":"56435df3f093cd92167c202d45bc91eec857be05e816cb91e6abd6bbaadf7941"} Sep 10 14:48:59.114051 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.113600 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfnbw" event={"ID":"3db175b6-6536-4b9c-854d-0630040d4d33","Type":"ContainerStarted","Data":"5c30dca9fdcc73b07301c5a3465dd28cd676d547b6f5608c30fa0b76283c37d1"} Sep 10 14:48:59.115563 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.115373 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" event={"ID":"41904cb2-4d32-486c-bf85-b6c601e61b12","Type":"ContainerStarted","Data":"bc6a448003d38f954fcdae51202375cd0fe22d518f937b02e641b8f4a6afc854"} Sep 10 14:48:59.133399 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.133346 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zfnbw" podStartSLOduration=3.581710647 podStartE2EDuration="6.133324109s" totalImagesPullingTime="2.47847543s" totalInitContainerRuntime="73.138032ms" isStatefulPod=true podCreationTimestamp="2026-09-10 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:59.13189363 +0000 UTC m=+77.946476373" watchObservedRunningTime="2026-09-10 14:48:59.133324109 +0000 UTC m=+77.947906863" Sep 10 14:48:59.925895 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.925866 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:48:59.931793 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.931772 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.934431 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.934407 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 14:48:59.934583 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.934565 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 14:48:59.934702 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.934681 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 14:48:59.934776 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.934762 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f2u0pk2t44u8g\"" Sep 10 14:48:59.935399 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935263 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 14:48:59.935399 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935309 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 14:48:59.935399 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935326 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 14:48:59.935399 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935336 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 14:48:59.935399 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935347 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 14:48:59.935670 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935470 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 14:48:59.935744 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935699 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 14:48:59.935827 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935790 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d6wjl\"" Sep 10 14:48:59.935927 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935860 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 14:48:59.935927 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935899 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 14:48:59.936022 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.935931 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 14:48:59.938060 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.938038 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 14:48:59.939886 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.939868 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 14:48:59.947757 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.947732 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:48:59.997045 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997011 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997045 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997047 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997272 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997081 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997272 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997105 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997272 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997186 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997272 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997272 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997259 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997285 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997322 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997355 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997432 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997456 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997497 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997485 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997735 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997547 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997735 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997587 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997735 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997633 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997735 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997662 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997735 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997687 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm46z\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-kube-api-access-vm46z\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997735 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997724 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:48:59.997990 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:48:59.997754 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098751 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098774 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098817 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098844 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098875 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098912 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098940 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098962 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.098994 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099018 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099042 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099085 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099110 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm46z\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-kube-api-access-vm46z\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.098699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099184 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.100150 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.100150 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.100150 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099267 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.100150 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.099849 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.100150 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.100013 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.100150 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.100003 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.100447 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.100195 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.103811 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.103392 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.103811 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.103552 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.105396 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.105371 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.108881 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.105997 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.108881 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.106449 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.108881 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.107227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.108881 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.108864 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.113081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.112523 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.113081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.112687 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.113081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.113042 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.113393 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.113374 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.113857 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.113829 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.115091 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.115023 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.115213 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.115183 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.116288 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.116249 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm46z\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-kube-api-access-vm46z\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.116649 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.116622 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.241817 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.241735 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:00.597175 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:00.597124 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:49:00.866991 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:49:00.866961 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1f9fba_ef50_40a6_8bbc_771c9b0ac583.slice/crio-a3a9b93e13f98b36e7712a9268683f782f99f33f65d41ada91bea8eee50728e1 WatchSource:0}: Error finding container a3a9b93e13f98b36e7712a9268683f782f99f33f65d41ada91bea8eee50728e1: Status 404 returned error can't find the container with id a3a9b93e13f98b36e7712a9268683f782f99f33f65d41ada91bea8eee50728e1 Sep 10 14:49:01.127302 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.127220 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerStarted","Data":"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17"} Sep 10 14:49:01.127302 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.127288 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerStarted","Data":"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8"} Sep 10 14:49:01.127702 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.127327 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerStarted","Data":"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db"} Sep 10 14:49:01.127702 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.127358 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerStarted","Data":"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e"} Sep 10 14:49:01.127702 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.127384 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerStarted","Data":"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45"} Sep 10 14:49:01.129330 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.129307 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" event={"ID":"3fa1eb81-b11f-4bc0-8794-9b05406dbc23","Type":"ContainerStarted","Data":"a6adb86f2deca6712f63db2cf282f4eff519943ff79bcefa90344f42da40a2dc"} Sep 10 14:49:01.129434 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.129361 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" event={"ID":"3fa1eb81-b11f-4bc0-8794-9b05406dbc23","Type":"ContainerStarted","Data":"4e45a5c6d3d775cc9b44343cdfaeb780e663574afcb56bad13ac7fb00fe765a3"} Sep 10 14:49:01.129434 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.129398 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" event={"ID":"3fa1eb81-b11f-4bc0-8794-9b05406dbc23","Type":"ContainerStarted","Data":"77f8c29eda7e109afdfe5c18f8e20b9dcf447e62a91c70d7a0bda9c5f8785512"} Sep 10 14:49:01.130729 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.130706 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" exitCode=0 Sep 10 14:49:01.130848 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.130792 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerDied","Data":"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884"} Sep 10 14:49:01.130906 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.130872 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerStarted","Data":"a3a9b93e13f98b36e7712a9268683f782f99f33f65d41ada91bea8eee50728e1"} Sep 10 14:49:01.132444 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.132421 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" event={"ID":"41904cb2-4d32-486c-bf85-b6c601e61b12","Type":"ContainerStarted","Data":"251bfb315e684627d62f362d4a813c426eb2a27b421a27d77df9f423b78ce99f"} Sep 10 14:49:01.132634 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.132615 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:49:01.137822 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.137791 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" Sep 10 14:49:01.168621 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:01.168579 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-s9n4d" podStartSLOduration=1.320437816 podStartE2EDuration="3.168565757s" totalImagesPullingTime="1.848127941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:01.167631199 +0000 UTC m=+79.982213953" watchObservedRunningTime="2026-09-10 14:49:01.168565757 +0000 UTC m=+79.983148505" Sep 10 14:49:02.142408 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:02.142368 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerStarted","Data":"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074"} Sep 10 14:49:02.146543 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:02.146512 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" event={"ID":"3fa1eb81-b11f-4bc0-8794-9b05406dbc23","Type":"ContainerStarted","Data":"c20f15f5b6b3fbfbebe64e3e4f1a31066d31928db48bff3f615c4cef07194e93"} Sep 10 14:49:02.146678 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:02.146579 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" event={"ID":"3fa1eb81-b11f-4bc0-8794-9b05406dbc23","Type":"ContainerStarted","Data":"2f37c89c9c6869775428140e30587ade5da037dad01d9b901054a263313f2e7e"} Sep 10 14:49:02.146678 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:02.146623 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" event={"ID":"3fa1eb81-b11f-4bc0-8794-9b05406dbc23","Type":"ContainerStarted","Data":"cb7b20826d253431dacbf1b48f621409661d6e37f48cb42f4a0d8478a16d622b"} Sep 10 14:49:02.165974 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:02.165854 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.113885015 podStartE2EDuration="8.165835797s" totalImagesPullingTime="5.035800382s" totalInitContainerRuntime="16.1504ms" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:02.165223455 +0000 UTC m=+80.979806207" watchObservedRunningTime="2026-09-10 14:49:02.165835797 +0000 UTC m=+80.980418551" Sep 10 14:49:02.192628 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:02.192568 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" podStartSLOduration=3.012212876 podStartE2EDuration="7.192552803s" totalImagesPullingTime="4.180339927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:02.191452678 +0000 UTC m=+81.006035447" watchObservedRunningTime="2026-09-10 14:49:02.192552803 +0000 UTC m=+81.007135554" Sep 10 14:49:02.288749 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:02.288720 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5"] Sep 10 14:49:03.151810 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:03.151768 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:49:05.163148 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:05.163107 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerStarted","Data":"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96"} Sep 10 14:49:05.163526 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:05.163187 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerStarted","Data":"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa"} Sep 10 14:49:05.163526 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:05.163223 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerStarted","Data":"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821"} Sep 10 14:49:05.163526 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:05.163251 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerStarted","Data":"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc"} Sep 10 14:49:05.163526 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:05.163274 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerStarted","Data":"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f"} Sep 10 14:49:05.163526 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:05.163308 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerStarted","Data":"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565"} Sep 10 14:49:05.200190 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:05.200139 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.861175318 podStartE2EDuration="6.200121742s" totalImagesPullingTime="3.318932008s" totalInitContainerRuntime="20.014416ms" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:05.192775857 +0000 UTC m=+84.007358609" watchObservedRunningTime="2026-09-10 14:49:05.200121742 +0000 UTC m=+84.014704493" Sep 10 14:49:05.242867 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:05.242835 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:49:09.161691 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:09.161665 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-764bd5c656-xb7f4" Sep 10 14:49:12.479273 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:12.479239 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7695b79b66-t8fvs"] Sep 10 14:49:25.084840 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:25.084786 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-msnqz" Sep 10 14:49:27.313934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.313876 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" podUID="28996ee4-e607-4806-a6f7-243770c7142d" containerName="registry" containerID="cri-o://9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8" gracePeriod=30 Sep 10 14:49:27.589953 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.589894 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:49:27.624909 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.624880 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-registry-certificates\") pod \"28996ee4-e607-4806-a6f7-243770c7142d\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " Sep 10 14:49:27.625045 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.624934 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-installation-pull-secrets\") pod \"28996ee4-e607-4806-a6f7-243770c7142d\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " Sep 10 14:49:27.625045 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.624990 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-image-registry-private-configuration\") pod \"28996ee4-e607-4806-a6f7-243770c7142d\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " Sep 10 14:49:27.625045 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.625038 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28996ee4-e607-4806-a6f7-243770c7142d-ca-trust-extracted\") pod \"28996ee4-e607-4806-a6f7-243770c7142d\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " Sep 10 14:49:27.625211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.625066 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-trusted-ca\") pod \"28996ee4-e607-4806-a6f7-243770c7142d\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " Sep 10 14:49:27.625211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.625099 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-bound-sa-token\") pod \"28996ee4-e607-4806-a6f7-243770c7142d\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " Sep 10 14:49:27.625211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.625128 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") pod \"28996ee4-e607-4806-a6f7-243770c7142d\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " Sep 10 14:49:27.625211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.625179 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xf8m\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-kube-api-access-2xf8m\") pod \"28996ee4-e607-4806-a6f7-243770c7142d\" (UID: \"28996ee4-e607-4806-a6f7-243770c7142d\") " Sep 10 14:49:27.625418 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.625389 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "28996ee4-e607-4806-a6f7-243770c7142d" (UID: "28996ee4-e607-4806-a6f7-243770c7142d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:49:27.625548 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.625484 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "28996ee4-e607-4806-a6f7-243770c7142d" (UID: "28996ee4-e607-4806-a6f7-243770c7142d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:49:27.628241 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.628066 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "28996ee4-e607-4806-a6f7-243770c7142d" (UID: "28996ee4-e607-4806-a6f7-243770c7142d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:49:27.628241 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.628216 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-kube-api-access-2xf8m" (OuterVolumeSpecName: "kube-api-access-2xf8m") pod "28996ee4-e607-4806-a6f7-243770c7142d" (UID: "28996ee4-e607-4806-a6f7-243770c7142d"). InnerVolumeSpecName "kube-api-access-2xf8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:49:27.628241 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.628108 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "28996ee4-e607-4806-a6f7-243770c7142d" (UID: "28996ee4-e607-4806-a6f7-243770c7142d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:49:27.628241 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.628179 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "28996ee4-e607-4806-a6f7-243770c7142d" (UID: "28996ee4-e607-4806-a6f7-243770c7142d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:49:27.628532 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.628288 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "28996ee4-e607-4806-a6f7-243770c7142d" (UID: "28996ee4-e607-4806-a6f7-243770c7142d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:49:27.634380 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.634356 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28996ee4-e607-4806-a6f7-243770c7142d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "28996ee4-e607-4806-a6f7-243770c7142d" (UID: "28996ee4-e607-4806-a6f7-243770c7142d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:49:27.727002 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.726973 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-registry-certificates\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:27.727126 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.727006 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-installation-pull-secrets\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:27.727126 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.727023 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28996ee4-e607-4806-a6f7-243770c7142d-image-registry-private-configuration\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:27.727126 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.727039 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28996ee4-e607-4806-a6f7-243770c7142d-ca-trust-extracted\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:27.727126 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.727060 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28996ee4-e607-4806-a6f7-243770c7142d-trusted-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:27.727126 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.727074 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-bound-sa-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:27.727126 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.727086 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-registry-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:27.727126 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:27.727098 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xf8m\" (UniqueName: \"kubernetes.io/projected/28996ee4-e607-4806-a6f7-243770c7142d-kube-api-access-2xf8m\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:28.257140 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.257104 2560 generic.go:356] "Generic (PLEG): container finished" podID="28996ee4-e607-4806-a6f7-243770c7142d" containerID="9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8" exitCode=0 Sep 10 14:49:28.257322 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.257158 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" Sep 10 14:49:28.257322 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.257198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" event={"ID":"28996ee4-e607-4806-a6f7-243770c7142d","Type":"ContainerDied","Data":"9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8"} Sep 10 14:49:28.257322 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.257285 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5" event={"ID":"28996ee4-e607-4806-a6f7-243770c7142d","Type":"ContainerDied","Data":"92133c02c3ea857d7ab43ecd4207bb2613bc8f7b9cc14954b1a6ad6a51f4eb65"} Sep 10 14:49:28.257322 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.257311 2560 scope.go:117] "RemoveContainer" containerID="9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8" Sep 10 14:49:28.274867 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.274842 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5"] Sep 10 14:49:28.280230 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.280209 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6ddbb4c9f5-q7mr5"] Sep 10 14:49:28.282165 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.282137 2560 scope.go:117] "RemoveContainer" containerID="9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8" Sep 10 14:49:28.282452 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:49:28.282432 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8\": container with ID starting with 9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8 not found: ID does not exist" containerID="9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8" Sep 10 14:49:28.282496 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:28.282467 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8"} err="failed to get container status \"9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8\": rpc error: code = NotFound desc = could not find container \"9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8\": container with ID starting with 9e8d87b2f43c8d3d4ef6ae9622362de1e73480007f00fea042b15e83a28898e8 not found: ID does not exist" Sep 10 14:49:29.663457 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:29.663426 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28996ee4-e607-4806-a6f7-243770c7142d" path="/var/lib/kubelet/pods/28996ee4-e607-4806-a6f7-243770c7142d/volumes" Sep 10 14:49:37.506475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.506414 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7695b79b66-t8fvs" podUID="dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" containerName="console" containerID="cri-o://ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b" gracePeriod=15 Sep 10 14:49:37.743850 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.743827 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:49:37.806737 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.806663 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-serving-cert\") pod \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " Sep 10 14:49:37.806737 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.806698 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-config\") pod \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " Sep 10 14:49:37.806737 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.806734 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-oauth-serving-cert\") pod \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " Sep 10 14:49:37.807018 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.806751 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-oauth-config\") pod \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " Sep 10 14:49:37.807018 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.806914 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-service-ca\") pod \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " Sep 10 14:49:37.807018 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.806999 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fjj2\" (UniqueName: \"kubernetes.io/projected/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-kube-api-access-6fjj2\") pod \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\" (UID: \"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2\") " Sep 10 14:49:37.807220 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.807178 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-config" (OuterVolumeSpecName: "console-config") pod "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" (UID: "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:49:37.807220 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.807183 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" (UID: "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:49:37.807381 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.807262 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-service-ca" (OuterVolumeSpecName: "service-ca") pod "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" (UID: "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:49:37.807381 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.807297 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-config\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:37.807381 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.807316 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-oauth-serving-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:37.809128 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.809101 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" (UID: "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:49:37.809217 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.809194 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" (UID: "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:49:37.809258 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.809231 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-kube-api-access-6fjj2" (OuterVolumeSpecName: "kube-api-access-6fjj2") pod "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" (UID: "dc50ae6e-d0a4-4765-b8ff-b921772e9cd2"). InnerVolumeSpecName "kube-api-access-6fjj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:49:37.908669 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.908636 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fjj2\" (UniqueName: \"kubernetes.io/projected/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-kube-api-access-6fjj2\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:37.908669 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.908664 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-serving-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:37.908669 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.908673 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-console-oauth-config\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:37.908879 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:37.908683 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:49:38.297476 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.297441 2560 generic.go:356] "Generic (PLEG): container finished" podID="dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" containerID="ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b" exitCode=0 Sep 10 14:49:38.297645 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.297481 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695b79b66-t8fvs" event={"ID":"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2","Type":"ContainerDied","Data":"ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b"} Sep 10 14:49:38.297645 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.297489 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695b79b66-t8fvs" Sep 10 14:49:38.297645 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.297516 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695b79b66-t8fvs" event={"ID":"dc50ae6e-d0a4-4765-b8ff-b921772e9cd2","Type":"ContainerDied","Data":"5789dacb54b04434f1922f3416fc96f47e1a74f8666fd1430c86c2f84c6ef1ed"} Sep 10 14:49:38.297645 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.297533 2560 scope.go:117] "RemoveContainer" containerID="ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b" Sep 10 14:49:38.311344 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.311281 2560 scope.go:117] "RemoveContainer" containerID="ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b" Sep 10 14:49:38.311770 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:49:38.311681 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b\": container with ID starting with ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b not found: ID does not exist" containerID="ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b" Sep 10 14:49:38.311770 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.311719 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b"} err="failed to get container status \"ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b\": rpc error: code = NotFound desc = could not find container \"ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b\": container with ID starting with ce433f96be06f92d4fbf9b8aebbc2ace2447679079ee825f930aad54ad934f8b not found: ID does not exist" Sep 10 14:49:38.324269 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.324248 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7695b79b66-t8fvs"] Sep 10 14:49:38.329953 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:38.329931 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7695b79b66-t8fvs"] Sep 10 14:49:39.663282 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:49:39.663248 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" path="/var/lib/kubelet/pods/dc50ae6e-d0a4-4765-b8ff-b921772e9cd2/volumes" Sep 10 14:50:00.242464 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:00.242435 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:00.261764 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:00.261737 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:00.392475 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:00.392449 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:13.909743 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:13.909701 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:50:13.910226 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:13.910088 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="alertmanager" containerID="cri-o://0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45" gracePeriod=120 Sep 10 14:50:13.910226 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:13.910159 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy-metric" containerID="cri-o://d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17" gracePeriod=120 Sep 10 14:50:13.910226 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:13.910185 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="config-reloader" containerID="cri-o://e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e" gracePeriod=120 Sep 10 14:50:13.910386 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:13.910212 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="prom-label-proxy" containerID="cri-o://712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074" gracePeriod=120 Sep 10 14:50:13.910386 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:13.910167 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy-web" containerID="cri-o://5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db" gracePeriod=120 Sep 10 14:50:13.910386 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:13.910213 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy" containerID="cri-o://afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8" gracePeriod=120 Sep 10 14:50:14.434514 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:14.434476 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerID="712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074" exitCode=0 Sep 10 14:50:14.434514 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:14.434504 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerID="afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8" exitCode=0 Sep 10 14:50:14.434514 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:14.434512 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerID="e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e" exitCode=0 Sep 10 14:50:14.434751 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:14.434527 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerID="0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45" exitCode=0 Sep 10 14:50:14.434751 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:14.434550 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerDied","Data":"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074"} Sep 10 14:50:14.434751 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:14.434638 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerDied","Data":"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8"} Sep 10 14:50:14.434751 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:14.434664 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerDied","Data":"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e"} Sep 10 14:50:14.434751 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:14.434689 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerDied","Data":"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45"} Sep 10 14:50:15.142159 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.142131 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.209305 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209273 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-main-db\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209450 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209343 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-tls-assets\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209450 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209375 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-cluster-tls-config\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209450 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209423 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-main-tls\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209450 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209448 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-metrics-client-ca\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209504 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-web\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209538 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-volume\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209563 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209590 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-trusted-ca-bundle\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209622 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-web-config\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.209666 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209652 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.212162 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209693 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-out\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.212162 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209742 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx96z\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-kube-api-access-mx96z\") pod \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\" (UID: \"6d54500a-fbcb-4889-8a5b-5c330efb9c94\") " Sep 10 14:50:15.213286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.209613 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:50:15.213286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.213221 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:15.213286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.212701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-volume" (OuterVolumeSpecName: "config-volume") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:15.213286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.212757 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-kube-api-access-mx96z" (OuterVolumeSpecName: "kube-api-access-mx96z") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "kube-api-access-mx96z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:50:15.213286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.212926 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:50:15.213286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.213182 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-out" (OuterVolumeSpecName: "config-out") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:50:15.213286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.213199 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:15.213617 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.213383 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:15.214734 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.214701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:15.216156 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.216115 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:15.216343 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.216321 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:15.219507 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.219483 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:15.224962 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.224940 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-web-config" (OuterVolumeSpecName: "web-config") pod "6d54500a-fbcb-4889-8a5b-5c330efb9c94" (UID: "6d54500a-fbcb-4889-8a5b-5c330efb9c94"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:15.310852 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310753 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-tls-assets\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.310852 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310778 2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-cluster-tls-config\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.310852 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310789 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-main-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.310852 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310821 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-metrics-client-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.310852 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310833 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.310852 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310843 2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-volume\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.310852 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310851 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.311147 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310859 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.311147 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310869 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-web-config\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.311147 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310877 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6d54500a-fbcb-4889-8a5b-5c330efb9c94-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.311147 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310886 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-config-out\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.311147 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310894 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx96z\" (UniqueName: \"kubernetes.io/projected/6d54500a-fbcb-4889-8a5b-5c330efb9c94-kube-api-access-mx96z\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.311147 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.310902 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6d54500a-fbcb-4889-8a5b-5c330efb9c94-alertmanager-main-db\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:15.441527 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.441495 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerID="d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17" exitCode=0 Sep 10 14:50:15.441527 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.441518 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerID="5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db" exitCode=0 Sep 10 14:50:15.441745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.441568 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerDied","Data":"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17"} Sep 10 14:50:15.441745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.441583 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.441745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.441632 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerDied","Data":"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db"} Sep 10 14:50:15.441745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.441660 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6d54500a-fbcb-4889-8a5b-5c330efb9c94","Type":"ContainerDied","Data":"3454c2d573f39111adf88043d1cadd29cfe8ee28d7671c68b92fb8dcad62833e"} Sep 10 14:50:15.441745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.441691 2560 scope.go:117] "RemoveContainer" containerID="712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074" Sep 10 14:50:15.448890 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.448873 2560 scope.go:117] "RemoveContainer" containerID="d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17" Sep 10 14:50:15.456481 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.456457 2560 scope.go:117] "RemoveContainer" containerID="afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8" Sep 10 14:50:15.462875 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.462859 2560 scope.go:117] "RemoveContainer" containerID="5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db" Sep 10 14:50:15.466009 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.465988 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:50:15.470084 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.470059 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:50:15.472753 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.472731 2560 scope.go:117] "RemoveContainer" containerID="e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e" Sep 10 14:50:15.480182 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.480167 2560 scope.go:117] "RemoveContainer" containerID="0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45" Sep 10 14:50:15.487354 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.487336 2560 scope.go:117] "RemoveContainer" containerID="ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5" Sep 10 14:50:15.493849 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.493828 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:50:15.494146 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494131 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28996ee4-e607-4806-a6f7-243770c7142d" containerName="registry" Sep 10 14:50:15.494217 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494149 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="28996ee4-e607-4806-a6f7-243770c7142d" containerName="registry" Sep 10 14:50:15.494217 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494160 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" containerName="console" Sep 10 14:50:15.494217 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494167 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" containerName="console" Sep 10 14:50:15.494217 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494182 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="init-config-reloader" Sep 10 14:50:15.494217 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494191 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="init-config-reloader" Sep 10 14:50:15.494217 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494206 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy-web" Sep 10 14:50:15.494217 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494214 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy-web" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494222 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494230 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494245 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="config-reloader" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494253 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="config-reloader" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494262 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy-metric" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494270 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy-metric" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494292 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="alertmanager" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494300 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="alertmanager" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494310 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="prom-label-proxy" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494319 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="prom-label-proxy" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494382 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy-web" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494394 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="28996ee4-e607-4806-a6f7-243770c7142d" containerName="registry" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494404 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494412 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="kube-rbac-proxy-metric" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494421 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc50ae6e-d0a4-4765-b8ff-b921772e9cd2" containerName="console" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494430 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="alertmanager" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494439 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="prom-label-proxy" Sep 10 14:50:15.494546 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494447 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" containerName="config-reloader" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494575 2560 scope.go:117] "RemoveContainer" containerID="712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:15.494864 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074\": container with ID starting with 712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074 not found: ID does not exist" containerID="712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494891 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074"} err="failed to get container status \"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074\": rpc error: code = NotFound desc = could not find container \"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074\": container with ID starting with 712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074 not found: ID does not exist" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.494908 2560 scope.go:117] "RemoveContainer" containerID="d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:15.495173 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17\": container with ID starting with d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17 not found: ID does not exist" containerID="d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.495196 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17"} err="failed to get container status \"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17\": rpc error: code = NotFound desc = could not find container \"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17\": container with ID starting with d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17 not found: ID does not exist" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.495221 2560 scope.go:117] "RemoveContainer" containerID="afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:15.495461 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8\": container with ID starting with afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8 not found: ID does not exist" containerID="afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8" Sep 10 14:50:15.495513 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.495480 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8"} err="failed to get container status \"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8\": rpc error: code = NotFound desc = could not find container \"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8\": container with ID starting with afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8 not found: ID does not exist" Sep 10 14:50:15.495964 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.495495 2560 scope.go:117] "RemoveContainer" containerID="5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db" Sep 10 14:50:15.495964 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:15.495763 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db\": container with ID starting with 5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db not found: ID does not exist" containerID="5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db" Sep 10 14:50:15.495964 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.495781 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db"} err="failed to get container status \"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db\": rpc error: code = NotFound desc = could not find container \"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db\": container with ID starting with 5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db not found: ID does not exist" Sep 10 14:50:15.495964 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.495809 2560 scope.go:117] "RemoveContainer" containerID="e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e" Sep 10 14:50:15.496136 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:15.496029 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e\": container with ID starting with e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e not found: ID does not exist" containerID="e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e" Sep 10 14:50:15.496136 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.496057 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e"} err="failed to get container status \"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e\": rpc error: code = NotFound desc = could not find container \"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e\": container with ID starting with e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e not found: ID does not exist" Sep 10 14:50:15.496136 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.496070 2560 scope.go:117] "RemoveContainer" containerID="0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45" Sep 10 14:50:15.496299 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:15.496280 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45\": container with ID starting with 0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45 not found: ID does not exist" containerID="0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45" Sep 10 14:50:15.496331 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.496302 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45"} err="failed to get container status \"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45\": rpc error: code = NotFound desc = could not find container \"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45\": container with ID starting with 0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45 not found: ID does not exist" Sep 10 14:50:15.496331 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.496313 2560 scope.go:117] "RemoveContainer" containerID="ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5" Sep 10 14:50:15.496538 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:15.496521 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5\": container with ID starting with ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5 not found: ID does not exist" containerID="ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5" Sep 10 14:50:15.496575 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.496542 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5"} err="failed to get container status \"ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5\": rpc error: code = NotFound desc = could not find container \"ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5\": container with ID starting with ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5 not found: ID does not exist" Sep 10 14:50:15.496575 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.496556 2560 scope.go:117] "RemoveContainer" containerID="712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074" Sep 10 14:50:15.496772 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.496753 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074"} err="failed to get container status \"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074\": rpc error: code = NotFound desc = could not find container \"712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074\": container with ID starting with 712abd68c2319b6997d23d78618ccd78aaea66281d7e4fce436c8bbacd3a4074 not found: ID does not exist" Sep 10 14:50:15.496876 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.496774 2560 scope.go:117] "RemoveContainer" containerID="d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17" Sep 10 14:50:15.497075 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497050 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17"} err="failed to get container status \"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17\": rpc error: code = NotFound desc = could not find container \"d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17\": container with ID starting with d62c9930afe2db7b3256b85edfc3a0a4940ea6ce9055644bd9e4264784b3de17 not found: ID does not exist" Sep 10 14:50:15.497075 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497074 2560 scope.go:117] "RemoveContainer" containerID="afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8" Sep 10 14:50:15.497302 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497285 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8"} err="failed to get container status \"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8\": rpc error: code = NotFound desc = could not find container \"afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8\": container with ID starting with afdc23a5cf3a9af71afd5c59db45bc120768a1ae651459aff3577fb77b2141e8 not found: ID does not exist" Sep 10 14:50:15.497343 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497303 2560 scope.go:117] "RemoveContainer" containerID="5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db" Sep 10 14:50:15.497532 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497516 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db"} err="failed to get container status \"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db\": rpc error: code = NotFound desc = could not find container \"5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db\": container with ID starting with 5fa631e44c62ea74f0bd4e84a9296102b8decbf6cc3815814a692c08967d80db not found: ID does not exist" Sep 10 14:50:15.497577 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497533 2560 scope.go:117] "RemoveContainer" containerID="e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e" Sep 10 14:50:15.497724 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497707 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e"} err="failed to get container status \"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e\": rpc error: code = NotFound desc = could not find container \"e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e\": container with ID starting with e01f385540c76c54d746ce4ce6ea014f35d752b44b216350baa6a3c71369213e not found: ID does not exist" Sep 10 14:50:15.497766 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497724 2560 scope.go:117] "RemoveContainer" containerID="0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45" Sep 10 14:50:15.497882 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497864 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.497972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497945 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45"} err="failed to get container status \"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45\": rpc error: code = NotFound desc = could not find container \"0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45\": container with ID starting with 0f05ec14d4ded63b51a73d867ac2c1f0c78a4f162c731527a3fd86d690a31c45 not found: ID does not exist" Sep 10 14:50:15.497972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.497967 2560 scope.go:117] "RemoveContainer" containerID="ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5" Sep 10 14:50:15.498424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.498393 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5"} err="failed to get container status \"ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5\": rpc error: code = NotFound desc = could not find container \"ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5\": container with ID starting with ecc922cdf003da8b3e95abd967f93bc4daa439b6fc7e897e91d9a6e281f8f6d5 not found: ID does not exist" Sep 10 14:50:15.500543 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.500524 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 14:50:15.500647 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.500560 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 14:50:15.500647 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.500539 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 14:50:15.500756 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.500676 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 14:50:15.500756 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.500697 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 14:50:15.500756 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.500714 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 14:50:15.500958 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.500941 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d2kgl\"" Sep 10 14:50:15.501017 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.500992 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 14:50:15.501120 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.501092 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 14:50:15.505574 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.505555 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 14:50:15.511972 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.511948 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512069 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.511982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-web-config\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512069 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512018 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512069 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512051 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512193 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512098 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91e1401f-0ccf-4b5c-85e3-672f09a45620-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512193 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512146 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512193 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91e1401f-0ccf-4b5c-85e3-672f09a45620-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512304 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512193 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512304 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512217 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91e1401f-0ccf-4b5c-85e3-672f09a45620-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512304 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512284 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91e1401f-0ccf-4b5c-85e3-672f09a45620-config-out\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512395 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512332 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91e1401f-0ccf-4b5c-85e3-672f09a45620-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512395 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512352 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-config-volume\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.512395 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.512373 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzss5\" (UniqueName: \"kubernetes.io/projected/91e1401f-0ccf-4b5c-85e3-672f09a45620-kube-api-access-bzss5\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.513509 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.513489 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:50:15.613785 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.613697 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.613785 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.613735 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91e1401f-0ccf-4b5c-85e3-672f09a45620-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.613785 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.613755 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614048 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.613843 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91e1401f-0ccf-4b5c-85e3-672f09a45620-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614048 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.613904 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91e1401f-0ccf-4b5c-85e3-672f09a45620-config-out\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614048 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.613948 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91e1401f-0ccf-4b5c-85e3-672f09a45620-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614048 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.613975 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-config-volume\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614259 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.614123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzss5\" (UniqueName: \"kubernetes.io/projected/91e1401f-0ccf-4b5c-85e3-672f09a45620-kube-api-access-bzss5\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614259 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.614196 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614259 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.614231 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-web-config\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614423 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.614260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614423 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.614296 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.614423 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.614327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91e1401f-0ccf-4b5c-85e3-672f09a45620-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.616085 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.614625 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91e1401f-0ccf-4b5c-85e3-672f09a45620-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.616085 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.614678 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91e1401f-0ccf-4b5c-85e3-672f09a45620-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.616085 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.615641 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91e1401f-0ccf-4b5c-85e3-672f09a45620-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.616837 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.616793 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.617004 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.616981 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91e1401f-0ccf-4b5c-85e3-672f09a45620-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.617066 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.617030 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.617131 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.617091 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-config-volume\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.617355 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.617338 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91e1401f-0ccf-4b5c-85e3-672f09a45620-config-out\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.617423 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.617357 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.617661 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.617643 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-web-config\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.618437 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.618420 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.618626 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.618610 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91e1401f-0ccf-4b5c-85e3-672f09a45620-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.625977 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.624092 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzss5\" (UniqueName: \"kubernetes.io/projected/91e1401f-0ccf-4b5c-85e3-672f09a45620-kube-api-access-bzss5\") pod \"alertmanager-main-0\" (UID: \"91e1401f-0ccf-4b5c-85e3-672f09a45620\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:15.663080 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.663059 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d54500a-fbcb-4889-8a5b-5c330efb9c94" path="/var/lib/kubelet/pods/6d54500a-fbcb-4889-8a5b-5c330efb9c94/volumes" Sep 10 14:50:15.807596 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:15.807563 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 14:50:16.084771 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:16.084743 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 14:50:16.087172 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:50:16.087147 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e1401f_0ccf_4b5c_85e3_672f09a45620.slice/crio-e64a00d989eb5fa0a8724db2d70fbe6a309ec4dcf7e05c621e6dbc9bbb2ea62c WatchSource:0}: Error finding container e64a00d989eb5fa0a8724db2d70fbe6a309ec4dcf7e05c621e6dbc9bbb2ea62c: Status 404 returned error can't find the container with id e64a00d989eb5fa0a8724db2d70fbe6a309ec4dcf7e05c621e6dbc9bbb2ea62c Sep 10 14:50:16.447034 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:16.446998 2560 generic.go:356] "Generic (PLEG): container finished" podID="91e1401f-0ccf-4b5c-85e3-672f09a45620" containerID="6e6f43762afad140d7804d9accddd3cbe979879eeee766eb682cd80ca2744e60" exitCode=0 Sep 10 14:50:16.447487 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:16.447091 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91e1401f-0ccf-4b5c-85e3-672f09a45620","Type":"ContainerDied","Data":"6e6f43762afad140d7804d9accddd3cbe979879eeee766eb682cd80ca2744e60"} Sep 10 14:50:16.447487 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:16.447155 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91e1401f-0ccf-4b5c-85e3-672f09a45620","Type":"ContainerStarted","Data":"e64a00d989eb5fa0a8724db2d70fbe6a309ec4dcf7e05c621e6dbc9bbb2ea62c"} Sep 10 14:50:17.455306 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:17.455273 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91e1401f-0ccf-4b5c-85e3-672f09a45620","Type":"ContainerStarted","Data":"3bdcfbd38404ddb1eb6d1b0792cc1fb278be03a26c7913f1afade69012d5afe2"} Sep 10 14:50:17.455663 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:17.455335 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91e1401f-0ccf-4b5c-85e3-672f09a45620","Type":"ContainerStarted","Data":"5c32befd63f8de58788fb7670a5a1c7ee6b372688393d39249efb26ad35393bc"} Sep 10 14:50:17.455663 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:17.455363 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91e1401f-0ccf-4b5c-85e3-672f09a45620","Type":"ContainerStarted","Data":"a4c710b1a836f37ce3d7f973ecfdcbf5a42a71b9b8204fb64b4876b003b1ab97"} Sep 10 14:50:17.455663 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:17.455387 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91e1401f-0ccf-4b5c-85e3-672f09a45620","Type":"ContainerStarted","Data":"73f965fd381ec3e0034420e8fb7965cd34d6c0bb8dc8da24b9e5ad120d224aef"} Sep 10 14:50:17.455663 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:17.455412 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91e1401f-0ccf-4b5c-85e3-672f09a45620","Type":"ContainerStarted","Data":"fd6618656067094b211cbb0b9ce29ded29242ca426343ece980a4024522c693e"} Sep 10 14:50:17.455663 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:17.455435 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91e1401f-0ccf-4b5c-85e3-672f09a45620","Type":"ContainerStarted","Data":"983ef3aa1e67df4571d157ee0a8a3d221071167cb11f74b9e72440d117f3e5d0"} Sep 10 14:50:17.481498 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:17.481441 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.467284411 podStartE2EDuration="2.481420364s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.135953ms" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:17.479997548 +0000 UTC m=+156.294580301" watchObservedRunningTime="2026-09-10 14:50:17.481420364 +0000 UTC m=+156.296003115" Sep 10 14:50:18.214791 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.214756 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:50:18.215089 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.215052 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="prometheus" containerID="cri-o://325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" gracePeriod=600 Sep 10 14:50:18.215181 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.215075 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy" containerID="cri-o://0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" gracePeriod=600 Sep 10 14:50:18.215181 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.215089 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" gracePeriod=600 Sep 10 14:50:18.215181 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.215107 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy-web" containerID="cri-o://177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" gracePeriod=600 Sep 10 14:50:18.215181 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.215115 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="thanos-sidecar" containerID="cri-o://514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" gracePeriod=600 Sep 10 14:50:18.215181 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.215126 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="config-reloader" containerID="cri-o://5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" gracePeriod=600 Sep 10 14:50:18.459469 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.459443 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:18.462207 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462174 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" exitCode=0 Sep 10 14:50:18.462207 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462196 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" exitCode=0 Sep 10 14:50:18.462207 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462202 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" exitCode=0 Sep 10 14:50:18.462207 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462208 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" exitCode=0 Sep 10 14:50:18.462207 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462214 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" exitCode=0 Sep 10 14:50:18.462432 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462219 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" exitCode=0 Sep 10 14:50:18.462432 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462243 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerDied","Data":"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96"} Sep 10 14:50:18.462432 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462331 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerDied","Data":"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa"} Sep 10 14:50:18.462432 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462359 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerDied","Data":"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821"} Sep 10 14:50:18.462432 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462379 2560 scope.go:117] "RemoveContainer" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" Sep 10 14:50:18.462432 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462390 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerDied","Data":"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc"} Sep 10 14:50:18.462432 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462416 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerDied","Data":"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f"} Sep 10 14:50:18.462718 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462439 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerDied","Data":"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565"} Sep 10 14:50:18.462718 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.462469 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583","Type":"ContainerDied","Data":"a3a9b93e13f98b36e7712a9268683f782f99f33f65d41ada91bea8eee50728e1"} Sep 10 14:50:18.471220 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.471183 2560 scope.go:117] "RemoveContainer" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" Sep 10 14:50:18.477992 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.477975 2560 scope.go:117] "RemoveContainer" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" Sep 10 14:50:18.486084 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.485789 2560 scope.go:117] "RemoveContainer" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" Sep 10 14:50:18.496456 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.496431 2560 scope.go:117] "RemoveContainer" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" Sep 10 14:50:18.504346 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.504327 2560 scope.go:117] "RemoveContainer" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" Sep 10 14:50:18.511372 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.511353 2560 scope.go:117] "RemoveContainer" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" Sep 10 14:50:18.518955 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.518934 2560 scope.go:117] "RemoveContainer" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" Sep 10 14:50:18.519244 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:18.519224 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": container with ID starting with 4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96 not found: ID does not exist" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" Sep 10 14:50:18.519286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.519254 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96"} err="failed to get container status \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": rpc error: code = NotFound desc = could not find container \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": container with ID starting with 4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96 not found: ID does not exist" Sep 10 14:50:18.519286 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.519275 2560 scope.go:117] "RemoveContainer" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" Sep 10 14:50:18.519521 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:18.519508 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": container with ID starting with 0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa not found: ID does not exist" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" Sep 10 14:50:18.519572 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.519524 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa"} err="failed to get container status \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": rpc error: code = NotFound desc = could not find container \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": container with ID starting with 0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa not found: ID does not exist" Sep 10 14:50:18.519572 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.519535 2560 scope.go:117] "RemoveContainer" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" Sep 10 14:50:18.519751 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:18.519735 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": container with ID starting with 177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821 not found: ID does not exist" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" Sep 10 14:50:18.519786 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.519755 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821"} err="failed to get container status \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": rpc error: code = NotFound desc = could not find container \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": container with ID starting with 177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821 not found: ID does not exist" Sep 10 14:50:18.519786 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.519768 2560 scope.go:117] "RemoveContainer" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" Sep 10 14:50:18.520063 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:18.520044 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": container with ID starting with 514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc not found: ID does not exist" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" Sep 10 14:50:18.520138 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.520072 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc"} err="failed to get container status \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": rpc error: code = NotFound desc = could not find container \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": container with ID starting with 514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc not found: ID does not exist" Sep 10 14:50:18.520138 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.520093 2560 scope.go:117] "RemoveContainer" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" Sep 10 14:50:18.520389 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:18.520369 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": container with ID starting with 5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f not found: ID does not exist" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" Sep 10 14:50:18.520462 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.520390 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f"} err="failed to get container status \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": rpc error: code = NotFound desc = could not find container \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": container with ID starting with 5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f not found: ID does not exist" Sep 10 14:50:18.520462 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.520405 2560 scope.go:117] "RemoveContainer" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" Sep 10 14:50:18.520679 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:18.520653 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": container with ID starting with 325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565 not found: ID does not exist" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" Sep 10 14:50:18.520731 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.520687 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565"} err="failed to get container status \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": rpc error: code = NotFound desc = could not find container \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": container with ID starting with 325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565 not found: ID does not exist" Sep 10 14:50:18.520731 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.520707 2560 scope.go:117] "RemoveContainer" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" Sep 10 14:50:18.521074 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:50:18.521051 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": container with ID starting with 657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884 not found: ID does not exist" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" Sep 10 14:50:18.521143 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.521080 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884"} err="failed to get container status \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": rpc error: code = NotFound desc = could not find container \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": container with ID starting with 657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884 not found: ID does not exist" Sep 10 14:50:18.521143 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.521098 2560 scope.go:117] "RemoveContainer" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" Sep 10 14:50:18.521363 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.521346 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96"} err="failed to get container status \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": rpc error: code = NotFound desc = could not find container \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": container with ID starting with 4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96 not found: ID does not exist" Sep 10 14:50:18.521363 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.521362 2560 scope.go:117] "RemoveContainer" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" Sep 10 14:50:18.521599 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.521584 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa"} err="failed to get container status \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": rpc error: code = NotFound desc = could not find container \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": container with ID starting with 0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa not found: ID does not exist" Sep 10 14:50:18.521660 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.521599 2560 scope.go:117] "RemoveContainer" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" Sep 10 14:50:18.521836 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.521820 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821"} err="failed to get container status \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": rpc error: code = NotFound desc = could not find container \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": container with ID starting with 177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821 not found: ID does not exist" Sep 10 14:50:18.521836 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.521835 2560 scope.go:117] "RemoveContainer" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" Sep 10 14:50:18.522043 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.522026 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc"} err="failed to get container status \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": rpc error: code = NotFound desc = could not find container \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": container with ID starting with 514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc not found: ID does not exist" Sep 10 14:50:18.522081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.522043 2560 scope.go:117] "RemoveContainer" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" Sep 10 14:50:18.522301 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.522282 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f"} err="failed to get container status \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": rpc error: code = NotFound desc = could not find container \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": container with ID starting with 5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f not found: ID does not exist" Sep 10 14:50:18.522378 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.522301 2560 scope.go:117] "RemoveContainer" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" Sep 10 14:50:18.522557 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.522538 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565"} err="failed to get container status \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": rpc error: code = NotFound desc = could not find container \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": container with ID starting with 325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565 not found: ID does not exist" Sep 10 14:50:18.522614 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.522557 2560 scope.go:117] "RemoveContainer" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" Sep 10 14:50:18.522817 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.522781 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884"} err="failed to get container status \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": rpc error: code = NotFound desc = could not find container \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": container with ID starting with 657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884 not found: ID does not exist" Sep 10 14:50:18.522886 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.522846 2560 scope.go:117] "RemoveContainer" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" Sep 10 14:50:18.523090 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.523074 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96"} err="failed to get container status \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": rpc error: code = NotFound desc = could not find container \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": container with ID starting with 4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96 not found: ID does not exist" Sep 10 14:50:18.523197 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.523090 2560 scope.go:117] "RemoveContainer" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" Sep 10 14:50:18.523335 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.523316 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa"} err="failed to get container status \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": rpc error: code = NotFound desc = could not find container \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": container with ID starting with 0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa not found: ID does not exist" Sep 10 14:50:18.523403 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.523338 2560 scope.go:117] "RemoveContainer" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" Sep 10 14:50:18.523584 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.523567 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821"} err="failed to get container status \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": rpc error: code = NotFound desc = could not find container \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": container with ID starting with 177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821 not found: ID does not exist" Sep 10 14:50:18.523634 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.523585 2560 scope.go:117] "RemoveContainer" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" Sep 10 14:50:18.523889 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.523865 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc"} err="failed to get container status \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": rpc error: code = NotFound desc = could not find container \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": container with ID starting with 514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc not found: ID does not exist" Sep 10 14:50:18.523889 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.523885 2560 scope.go:117] "RemoveContainer" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" Sep 10 14:50:18.524117 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.524099 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f"} err="failed to get container status \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": rpc error: code = NotFound desc = could not find container \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": container with ID starting with 5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f not found: ID does not exist" Sep 10 14:50:18.524188 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.524119 2560 scope.go:117] "RemoveContainer" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" Sep 10 14:50:18.524381 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.524364 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565"} err="failed to get container status \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": rpc error: code = NotFound desc = could not find container \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": container with ID starting with 325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565 not found: ID does not exist" Sep 10 14:50:18.524447 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.524382 2560 scope.go:117] "RemoveContainer" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" Sep 10 14:50:18.524596 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.524579 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884"} err="failed to get container status \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": rpc error: code = NotFound desc = could not find container \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": container with ID starting with 657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884 not found: ID does not exist" Sep 10 14:50:18.524664 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.524596 2560 scope.go:117] "RemoveContainer" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" Sep 10 14:50:18.524862 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.524845 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96"} err="failed to get container status \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": rpc error: code = NotFound desc = could not find container \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": container with ID starting with 4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96 not found: ID does not exist" Sep 10 14:50:18.524934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.524862 2560 scope.go:117] "RemoveContainer" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" Sep 10 14:50:18.525101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.525084 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa"} err="failed to get container status \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": rpc error: code = NotFound desc = could not find container \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": container with ID starting with 0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa not found: ID does not exist" Sep 10 14:50:18.525157 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.525110 2560 scope.go:117] "RemoveContainer" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" Sep 10 14:50:18.525348 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.525330 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821"} err="failed to get container status \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": rpc error: code = NotFound desc = could not find container \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": container with ID starting with 177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821 not found: ID does not exist" Sep 10 14:50:18.525411 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.525350 2560 scope.go:117] "RemoveContainer" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" Sep 10 14:50:18.525566 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.525545 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc"} err="failed to get container status \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": rpc error: code = NotFound desc = could not find container \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": container with ID starting with 514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc not found: ID does not exist" Sep 10 14:50:18.525606 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.525567 2560 scope.go:117] "RemoveContainer" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" Sep 10 14:50:18.525779 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.525762 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f"} err="failed to get container status \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": rpc error: code = NotFound desc = could not find container \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": container with ID starting with 5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f not found: ID does not exist" Sep 10 14:50:18.525779 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.525778 2560 scope.go:117] "RemoveContainer" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" Sep 10 14:50:18.526084 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526065 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565"} err="failed to get container status \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": rpc error: code = NotFound desc = could not find container \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": container with ID starting with 325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565 not found: ID does not exist" Sep 10 14:50:18.526084 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526083 2560 scope.go:117] "RemoveContainer" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" Sep 10 14:50:18.526315 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526297 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884"} err="failed to get container status \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": rpc error: code = NotFound desc = could not find container \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": container with ID starting with 657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884 not found: ID does not exist" Sep 10 14:50:18.526315 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526314 2560 scope.go:117] "RemoveContainer" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" Sep 10 14:50:18.526514 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526499 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96"} err="failed to get container status \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": rpc error: code = NotFound desc = could not find container \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": container with ID starting with 4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96 not found: ID does not exist" Sep 10 14:50:18.526578 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526513 2560 scope.go:117] "RemoveContainer" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" Sep 10 14:50:18.526719 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526702 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa"} err="failed to get container status \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": rpc error: code = NotFound desc = could not find container \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": container with ID starting with 0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa not found: ID does not exist" Sep 10 14:50:18.526776 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526720 2560 scope.go:117] "RemoveContainer" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" Sep 10 14:50:18.526944 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526927 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821"} err="failed to get container status \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": rpc error: code = NotFound desc = could not find container \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": container with ID starting with 177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821 not found: ID does not exist" Sep 10 14:50:18.527007 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.526944 2560 scope.go:117] "RemoveContainer" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" Sep 10 14:50:18.527137 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.527116 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc"} err="failed to get container status \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": rpc error: code = NotFound desc = could not find container \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": container with ID starting with 514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc not found: ID does not exist" Sep 10 14:50:18.527197 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.527137 2560 scope.go:117] "RemoveContainer" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" Sep 10 14:50:18.527338 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.527322 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f"} err="failed to get container status \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": rpc error: code = NotFound desc = could not find container \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": container with ID starting with 5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f not found: ID does not exist" Sep 10 14:50:18.527393 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.527337 2560 scope.go:117] "RemoveContainer" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" Sep 10 14:50:18.527525 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.527510 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565"} err="failed to get container status \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": rpc error: code = NotFound desc = could not find container \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": container with ID starting with 325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565 not found: ID does not exist" Sep 10 14:50:18.527580 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.527525 2560 scope.go:117] "RemoveContainer" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" Sep 10 14:50:18.527809 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.527774 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884"} err="failed to get container status \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": rpc error: code = NotFound desc = could not find container \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": container with ID starting with 657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884 not found: ID does not exist" Sep 10 14:50:18.527868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.527809 2560 scope.go:117] "RemoveContainer" containerID="4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96" Sep 10 14:50:18.528037 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528013 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96"} err="failed to get container status \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": rpc error: code = NotFound desc = could not find container \"4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96\": container with ID starting with 4000364e489b6503104e7dd1c7549eddc717f8c72483a2ecd8792fde9ecb5d96 not found: ID does not exist" Sep 10 14:50:18.528093 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528037 2560 scope.go:117] "RemoveContainer" containerID="0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa" Sep 10 14:50:18.528252 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528237 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa"} err="failed to get container status \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": rpc error: code = NotFound desc = could not find container \"0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa\": container with ID starting with 0a1d0dd2bd4ea64158250ee323c738fa7071fe492c182024fc7a5cc386d64afa not found: ID does not exist" Sep 10 14:50:18.528314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528251 2560 scope.go:117] "RemoveContainer" containerID="177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821" Sep 10 14:50:18.528452 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528435 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821"} err="failed to get container status \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": rpc error: code = NotFound desc = could not find container \"177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821\": container with ID starting with 177c71a2073fcd73d439ff5fa7abbf4acfac9321d32edd8192dff20faccf9821 not found: ID does not exist" Sep 10 14:50:18.528510 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528453 2560 scope.go:117] "RemoveContainer" containerID="514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc" Sep 10 14:50:18.528661 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528643 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc"} err="failed to get container status \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": rpc error: code = NotFound desc = could not find container \"514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc\": container with ID starting with 514e6f044608836a310983f0826d7a706f539befc970e49cb16b87d54d9fd4cc not found: ID does not exist" Sep 10 14:50:18.528716 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528662 2560 scope.go:117] "RemoveContainer" containerID="5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f" Sep 10 14:50:18.528928 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528911 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f"} err="failed to get container status \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": rpc error: code = NotFound desc = could not find container \"5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f\": container with ID starting with 5fd6d8c299b5c7bdb7b03b2bc63410e38c65491d6cce7ec0d3b2811cdf35631f not found: ID does not exist" Sep 10 14:50:18.528999 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.528929 2560 scope.go:117] "RemoveContainer" containerID="325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565" Sep 10 14:50:18.529165 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.529146 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565"} err="failed to get container status \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": rpc error: code = NotFound desc = could not find container \"325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565\": container with ID starting with 325e84b7ec021b97a3b04f63163eccb78d50e4c5a8713d8344d84775b4bff565 not found: ID does not exist" Sep 10 14:50:18.529225 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.529167 2560 scope.go:117] "RemoveContainer" containerID="657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884" Sep 10 14:50:18.529399 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.529382 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884"} err="failed to get container status \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": rpc error: code = NotFound desc = could not find container \"657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884\": container with ID starting with 657be816070e77b2aba5dd91bb5ab68c630b2ecf4a3c66f420cf65e7c9048884 not found: ID does not exist" Sep 10 14:50:18.544744 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.544717 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.544876 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.544773 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-trusted-ca-bundle\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.544876 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.544834 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-0\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.544876 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.544862 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545038 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.544891 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545038 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.544917 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-serving-certs-ca-bundle\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545038 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.544948 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-kubelet-serving-ca-bundle\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545038 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.544986 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-1\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545038 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545022 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-web-config\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545269 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545058 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-metrics-client-ca\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545269 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545108 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-grpc-tls\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545269 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545132 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-metrics-client-certs\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545269 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545157 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-2\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545269 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545181 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-tls-assets\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545269 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545234 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-tls\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545269 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545268 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-kube-rbac-proxy\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545588 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545230 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:18.545588 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545335 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-thanos-prometheus-http-client-file\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545588 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545361 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm46z\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-kube-api-access-vm46z\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545588 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545392 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-db\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545588 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545418 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config-out\") pod \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\" (UID: \"0f1f9fba-ef50-40a6-8bbc-771c9b0ac583\") " Sep 10 14:50:18.545879 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545846 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:18.545937 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.545882 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:18.546318 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.546288 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:18.546668 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.546643 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:18.547099 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.547068 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:50:18.547424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.547161 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:18.547877 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.547639 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:50:18.548056 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.547992 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548408 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548430 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-db\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548449 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548465 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548481 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548496 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548508 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548521 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.548868 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.548534 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-configmap-metrics-client-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.549751 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.549718 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.550371 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.550229 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-kube-api-access-vm46z" (OuterVolumeSpecName: "kube-api-access-vm46z") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "kube-api-access-vm46z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:50:18.550371 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.550320 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config-out" (OuterVolumeSpecName: "config-out") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:50:18.550371 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.550362 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.550539 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.550401 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.550676 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.550653 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.550760 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.550733 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config" (OuterVolumeSpecName: "config") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.550928 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.550847 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.550928 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.550909 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:50:18.551600 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.551580 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.559113 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.559093 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-web-config" (OuterVolumeSpecName: "web-config") pod "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" (UID: "0f1f9fba-ef50-40a6-8bbc-771c9b0ac583"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:50:18.649618 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649588 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-web-config\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649618 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649614 2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-grpc-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649627 2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-metrics-client-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649641 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-tls-assets\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649652 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649663 2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-kube-rbac-proxy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649677 2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649690 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm46z\" (UniqueName: \"kubernetes.io/projected/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-kube-api-access-vm46z\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649701 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config-out\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649713 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:18.649792 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:18.649725 2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583-config\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 14:50:19.467124 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.467089 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.489942 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.489883 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:50:19.494194 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.494171 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:50:19.519770 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.519744 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:50:19.520035 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520021 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy" Sep 10 14:50:19.520081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520036 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy" Sep 10 14:50:19.520081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520048 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy-web" Sep 10 14:50:19.520081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520053 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy-web" Sep 10 14:50:19.520081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520065 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="prometheus" Sep 10 14:50:19.520081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520070 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="prometheus" Sep 10 14:50:19.520081 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520079 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="config-reloader" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520084 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="config-reloader" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520090 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy-thanos" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520095 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy-thanos" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520102 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="thanos-sidecar" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520107 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="thanos-sidecar" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520115 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="init-config-reloader" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520120 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="init-config-reloader" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520160 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy-web" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520166 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="thanos-sidecar" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520173 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="config-reloader" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520178 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520185 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="kube-rbac-proxy-thanos" Sep 10 14:50:19.520276 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.520192 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" containerName="prometheus" Sep 10 14:50:19.523946 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.523931 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.526670 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.526652 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 14:50:19.526866 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.526852 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f2u0pk2t44u8g\"" Sep 10 14:50:19.527237 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.527222 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 14:50:19.527307 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.527269 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 14:50:19.527397 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.527383 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 14:50:19.527441 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.527429 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 14:50:19.527825 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.527793 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 14:50:19.527897 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.527834 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 14:50:19.528427 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.528259 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 14:50:19.528427 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.528282 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d6wjl\"" Sep 10 14:50:19.528427 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.528297 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 14:50:19.528427 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.528268 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 14:50:19.528427 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.528307 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 14:50:19.528758 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.528440 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 14:50:19.528758 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.528542 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 14:50:19.529900 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.529880 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 14:50:19.533709 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.533690 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 14:50:19.539422 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.539401 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:50:19.555970 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.555944 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556060 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.555971 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556060 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.555990 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556060 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556006 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4k6t\" (UniqueName: \"kubernetes.io/projected/cc3da546-573b-48e7-996a-23d87c252b44-kube-api-access-q4k6t\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556060 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556031 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556192 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556081 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556192 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556113 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556192 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556137 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556192 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556155 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc3da546-573b-48e7-996a-23d87c252b44-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556192 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556177 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556354 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556218 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556354 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556259 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556354 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556354 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556302 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556354 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556346 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-config\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556559 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556365 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc3da546-573b-48e7-996a-23d87c252b44-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556559 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556386 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556559 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556407 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556559 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556442 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.556559 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.556477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.657459 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657431 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.657567 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.657567 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657501 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.657567 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-config\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.657567 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657551 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc3da546-573b-48e7-996a-23d87c252b44-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.657745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657574 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.657745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657597 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.657745 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.658034 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657772 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.658034 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.657830 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658300 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658545 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658608 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658641 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658668 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4k6t\" (UniqueName: \"kubernetes.io/projected/cc3da546-573b-48e7-996a-23d87c252b44-kube-api-access-q4k6t\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658700 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658730 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658734 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658760 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658816 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658884 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc3da546-573b-48e7-996a-23d87c252b44-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658910 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.659050 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.658941 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.660072 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.660045 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.660594 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.660273 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.660988 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.660970 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.661646 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.661171 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc3da546-573b-48e7-996a-23d87c252b44-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.661646 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.661422 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-config\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.661646 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.661463 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.661646 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.661612 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.662760 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.662735 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.663025 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.662975 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.663130 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.663118 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.664099 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.664076 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.664503 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.664485 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.664716 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.664695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc3da546-573b-48e7-996a-23d87c252b44-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.664788 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.664770 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc3da546-573b-48e7-996a-23d87c252b44-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.664950 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.664929 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.665009 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.664998 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc3da546-573b-48e7-996a-23d87c252b44-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.666164 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.666139 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1f9fba-ef50-40a6-8bbc-771c9b0ac583" path="/var/lib/kubelet/pods/0f1f9fba-ef50-40a6-8bbc-771c9b0ac583/volumes" Sep 10 14:50:19.667106 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.667088 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4k6t\" (UniqueName: \"kubernetes.io/projected/cc3da546-573b-48e7-996a-23d87c252b44-kube-api-access-q4k6t\") pod \"prometheus-k8s-0\" (UID: \"cc3da546-573b-48e7-996a-23d87c252b44\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:19.833684 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:19.833631 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:50:20.110133 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:20.110066 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 14:50:20.112965 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:50:20.112939 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3da546_573b_48e7_996a_23d87c252b44.slice/crio-1afbe584b05878fe67ec53163a576c2795734223ad8bdf35ace75310f0ba41b7 WatchSource:0}: Error finding container 1afbe584b05878fe67ec53163a576c2795734223ad8bdf35ace75310f0ba41b7: Status 404 returned error can't find the container with id 1afbe584b05878fe67ec53163a576c2795734223ad8bdf35ace75310f0ba41b7 Sep 10 14:50:20.477741 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:20.477706 2560 generic.go:356] "Generic (PLEG): container finished" podID="cc3da546-573b-48e7-996a-23d87c252b44" containerID="89d6c20ad239b47345d6a6ff505ddf736565a91774725007d44361934795999d" exitCode=0 Sep 10 14:50:20.478087 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:20.477788 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc3da546-573b-48e7-996a-23d87c252b44","Type":"ContainerDied","Data":"89d6c20ad239b47345d6a6ff505ddf736565a91774725007d44361934795999d"} Sep 10 14:50:20.478087 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:20.477891 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc3da546-573b-48e7-996a-23d87c252b44","Type":"ContainerStarted","Data":"1afbe584b05878fe67ec53163a576c2795734223ad8bdf35ace75310f0ba41b7"} Sep 10 14:50:21.484843 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:21.484809 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc3da546-573b-48e7-996a-23d87c252b44","Type":"ContainerStarted","Data":"230f4e84eb1d20659a955cc9fd7d80b02ae72dfe7bce9ce33e6e36dcebd82f63"} Sep 10 14:50:21.485237 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:21.484872 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc3da546-573b-48e7-996a-23d87c252b44","Type":"ContainerStarted","Data":"c4fd36027bf5f2fb545eaf7917ffc1c25cabaac15d60e3d0f6be8410c66e0012"} Sep 10 14:50:21.485237 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:21.484901 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc3da546-573b-48e7-996a-23d87c252b44","Type":"ContainerStarted","Data":"8559b0acea6e118fc8d6293251d9c23ea1b0f2c22921c52e2eec5333651edc1d"} Sep 10 14:50:21.485237 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:21.484925 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc3da546-573b-48e7-996a-23d87c252b44","Type":"ContainerStarted","Data":"067dd55807704430d3193f0a2a1c2e2f0f84cdae1c13f37872ab82233e2c762b"} Sep 10 14:50:21.485237 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:21.484948 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc3da546-573b-48e7-996a-23d87c252b44","Type":"ContainerStarted","Data":"84e21b50784947f8d8ef252c1233180c5a5f52acb0020152217eb62f38ce6145"} Sep 10 14:50:21.485237 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:21.484971 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc3da546-573b-48e7-996a-23d87c252b44","Type":"ContainerStarted","Data":"10b36cfa301e72d8259a37e7703a6ff1a10fca00e0ab20adc9883cd6e4538652"} Sep 10 14:50:21.516527 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:21.516468 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.498355282 podStartE2EDuration="2.516448603s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.093321ms" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:21.51529538 +0000 UTC m=+160.329878131" watchObservedRunningTime="2026-09-10 14:50:21.516448603 +0000 UTC m=+160.331031356" Sep 10 14:50:24.834448 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:50:24.834413 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:51:19.834086 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:51:19.834052 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:51:19.849079 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:51:19.849055 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:51:20.731842 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:51:20.731814 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 14:52:41.582997 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:41.582959 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:52:41.585446 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:41.585421 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:52:41.588960 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:41.588941 2560 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 14:52:53.522109 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.522077 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vzsxz"] Sep 10 14:52:53.524939 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.524923 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:53.527485 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.527463 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 14:52:53.527577 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.527516 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4bzvs\"" Sep 10 14:52:53.528284 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.528269 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 14:52:53.534502 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.534479 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vzsxz"] Sep 10 14:52:53.632159 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.632123 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4708b9c-5223-40f2-9d20-180d9998712b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vzsxz\" (UID: \"d4708b9c-5223-40f2-9d20-180d9998712b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:53.632353 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.632171 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrzb\" (UniqueName: \"kubernetes.io/projected/d4708b9c-5223-40f2-9d20-180d9998712b-kube-api-access-qrrzb\") pod \"cert-manager-webhook-6486ffd997-vzsxz\" (UID: \"d4708b9c-5223-40f2-9d20-180d9998712b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:53.732974 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.732940 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4708b9c-5223-40f2-9d20-180d9998712b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vzsxz\" (UID: \"d4708b9c-5223-40f2-9d20-180d9998712b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:53.733134 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.732986 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrzb\" (UniqueName: \"kubernetes.io/projected/d4708b9c-5223-40f2-9d20-180d9998712b-kube-api-access-qrrzb\") pod \"cert-manager-webhook-6486ffd997-vzsxz\" (UID: \"d4708b9c-5223-40f2-9d20-180d9998712b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:53.745066 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.745043 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4708b9c-5223-40f2-9d20-180d9998712b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vzsxz\" (UID: \"d4708b9c-5223-40f2-9d20-180d9998712b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:53.748123 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.748097 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrzb\" (UniqueName: \"kubernetes.io/projected/d4708b9c-5223-40f2-9d20-180d9998712b-kube-api-access-qrrzb\") pod \"cert-manager-webhook-6486ffd997-vzsxz\" (UID: \"d4708b9c-5223-40f2-9d20-180d9998712b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:53.846657 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:53.846599 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:54.120295 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:54.120268 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vzsxz"] Sep 10 14:52:54.123281 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:52:54.123250 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4708b9c_5223_40f2_9d20_180d9998712b.slice/crio-6f79e0643c2e4709efe0158d46ede304c477cdc0ddf301dc879fd8a7fea13c54 WatchSource:0}: Error finding container 6f79e0643c2e4709efe0158d46ede304c477cdc0ddf301dc879fd8a7fea13c54: Status 404 returned error can't find the container with id 6f79e0643c2e4709efe0158d46ede304c477cdc0ddf301dc879fd8a7fea13c54 Sep 10 14:52:54.125254 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:54.125232 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:52:55.069273 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:55.069240 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" event={"ID":"d4708b9c-5223-40f2-9d20-180d9998712b","Type":"ContainerStarted","Data":"6f79e0643c2e4709efe0158d46ede304c477cdc0ddf301dc879fd8a7fea13c54"} Sep 10 14:52:58.084152 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:58.084113 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" event={"ID":"d4708b9c-5223-40f2-9d20-180d9998712b","Type":"ContainerStarted","Data":"17e3eeac5d50e8c917ef09b398a15142f414fbf169950b51cdde699dfbb57ca5"} Sep 10 14:52:58.084519 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:58.084177 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:52:58.101748 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:52:58.101702 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" podStartSLOduration=1.356351821 podStartE2EDuration="5.101688484s" totalImagesPullingTime="3.745336663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:52:58.100846981 +0000 UTC m=+316.915429733" watchObservedRunningTime="2026-09-10 14:52:58.101688484 +0000 UTC m=+316.916271235" Sep 10 14:53:04.089513 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:04.089477 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vzsxz" Sep 10 14:53:14.004715 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.004680 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9p6bt"] Sep 10 14:53:14.007447 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.007428 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9p6bt" Sep 10 14:53:14.010049 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.010023 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2btlw\"" Sep 10 14:53:14.017304 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.017284 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9p6bt"] Sep 10 14:53:14.100193 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.100165 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a6ee58-befa-4343-8e9a-1bf9ac466624-bound-sa-token\") pod \"cert-manager-55b4d954d5-9p6bt\" (UID: \"50a6ee58-befa-4343-8e9a-1bf9ac466624\") " pod="cert-manager/cert-manager-55b4d954d5-9p6bt" Sep 10 14:53:14.100324 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.100209 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p72wp\" (UniqueName: \"kubernetes.io/projected/50a6ee58-befa-4343-8e9a-1bf9ac466624-kube-api-access-p72wp\") pod \"cert-manager-55b4d954d5-9p6bt\" (UID: \"50a6ee58-befa-4343-8e9a-1bf9ac466624\") " pod="cert-manager/cert-manager-55b4d954d5-9p6bt" Sep 10 14:53:14.201043 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.201017 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a6ee58-befa-4343-8e9a-1bf9ac466624-bound-sa-token\") pod \"cert-manager-55b4d954d5-9p6bt\" (UID: \"50a6ee58-befa-4343-8e9a-1bf9ac466624\") " pod="cert-manager/cert-manager-55b4d954d5-9p6bt" Sep 10 14:53:14.201211 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.201055 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p72wp\" (UniqueName: \"kubernetes.io/projected/50a6ee58-befa-4343-8e9a-1bf9ac466624-kube-api-access-p72wp\") pod \"cert-manager-55b4d954d5-9p6bt\" (UID: \"50a6ee58-befa-4343-8e9a-1bf9ac466624\") " pod="cert-manager/cert-manager-55b4d954d5-9p6bt" Sep 10 14:53:14.209493 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.209465 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a6ee58-befa-4343-8e9a-1bf9ac466624-bound-sa-token\") pod \"cert-manager-55b4d954d5-9p6bt\" (UID: \"50a6ee58-befa-4343-8e9a-1bf9ac466624\") " pod="cert-manager/cert-manager-55b4d954d5-9p6bt" Sep 10 14:53:14.209700 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.209677 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p72wp\" (UniqueName: \"kubernetes.io/projected/50a6ee58-befa-4343-8e9a-1bf9ac466624-kube-api-access-p72wp\") pod \"cert-manager-55b4d954d5-9p6bt\" (UID: \"50a6ee58-befa-4343-8e9a-1bf9ac466624\") " pod="cert-manager/cert-manager-55b4d954d5-9p6bt" Sep 10 14:53:14.316494 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.316422 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9p6bt" Sep 10 14:53:14.582259 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:14.582112 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9p6bt"] Sep 10 14:53:14.584736 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:53:14.584703 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50a6ee58_befa_4343_8e9a_1bf9ac466624.slice/crio-36be35c88b3d5b6a96e7fb7b44950a2fc6821ab9e6ebd28796d6c818f397de83 WatchSource:0}: Error finding container 36be35c88b3d5b6a96e7fb7b44950a2fc6821ab9e6ebd28796d6c818f397de83: Status 404 returned error can't find the container with id 36be35c88b3d5b6a96e7fb7b44950a2fc6821ab9e6ebd28796d6c818f397de83 Sep 10 14:53:15.153280 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:15.153241 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9p6bt" event={"ID":"50a6ee58-befa-4343-8e9a-1bf9ac466624","Type":"ContainerStarted","Data":"29db59b7627d9a00d36f11586c26c32843c6d6ef0e280227c1509a14b2ee4852"} Sep 10 14:53:15.153280 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:15.153281 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9p6bt" event={"ID":"50a6ee58-befa-4343-8e9a-1bf9ac466624","Type":"ContainerStarted","Data":"36be35c88b3d5b6a96e7fb7b44950a2fc6821ab9e6ebd28796d6c818f397de83"} Sep 10 14:53:15.167925 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:15.167878 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9p6bt" podStartSLOduration=2.167864592 podStartE2EDuration="2.167864592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:15.16757417 +0000 UTC m=+333.982156940" watchObservedRunningTime="2026-09-10 14:53:15.167864592 +0000 UTC m=+333.982447343" Sep 10 14:53:26.689742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.689700 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr"] Sep 10 14:53:26.693599 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.693566 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.696280 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.696250 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 14:53:26.696410 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.696290 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 14:53:26.696410 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.696319 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 14:53:26.696542 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.696528 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 14:53:26.696737 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.696722 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-89wd6\"" Sep 10 14:53:26.710098 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.710080 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr"] Sep 10 14:53:26.784911 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.784877 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04df52ba-2f52-4ea1-9b82-9b562cc362c1-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.785074 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.784931 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04df52ba-2f52-4ea1-9b82-9b562cc362c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.785074 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.784955 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04df52ba-2f52-4ea1-9b82-9b562cc362c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.785074 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.785029 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzpw\" (UniqueName: \"kubernetes.io/projected/04df52ba-2f52-4ea1-9b82-9b562cc362c1-kube-api-access-kgzpw\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.785183 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.785091 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04df52ba-2f52-4ea1-9b82-9b562cc362c1-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.885994 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.885961 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04df52ba-2f52-4ea1-9b82-9b562cc362c1-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.886134 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.886002 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04df52ba-2f52-4ea1-9b82-9b562cc362c1-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.886134 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.886117 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04df52ba-2f52-4ea1-9b82-9b562cc362c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.886233 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.886146 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04df52ba-2f52-4ea1-9b82-9b562cc362c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.886233 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.886202 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzpw\" (UniqueName: \"kubernetes.io/projected/04df52ba-2f52-4ea1-9b82-9b562cc362c1-kube-api-access-kgzpw\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.886334 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.886306 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/04df52ba-2f52-4ea1-9b82-9b562cc362c1-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.888242 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.888208 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04df52ba-2f52-4ea1-9b82-9b562cc362c1-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.888603 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.888584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04df52ba-2f52-4ea1-9b82-9b562cc362c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.888648 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.888593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/04df52ba-2f52-4ea1-9b82-9b562cc362c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:26.893672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:26.893654 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzpw\" (UniqueName: \"kubernetes.io/projected/04df52ba-2f52-4ea1-9b82-9b562cc362c1-kube-api-access-kgzpw\") pod \"opendatahub-operator-controller-manager-759746987c-x6spr\" (UID: \"04df52ba-2f52-4ea1-9b82-9b562cc362c1\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:27.003901 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:27.003824 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:27.288931 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:27.288893 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr"] Sep 10 14:53:27.291483 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:53:27.291450 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04df52ba_2f52_4ea1_9b82_9b562cc362c1.slice/crio-aa16f69c5dbd546af0287c7aba6819315859b9967ca20736eac34128fa6e4af9 WatchSource:0}: Error finding container aa16f69c5dbd546af0287c7aba6819315859b9967ca20736eac34128fa6e4af9: Status 404 returned error can't find the container with id aa16f69c5dbd546af0287c7aba6819315859b9967ca20736eac34128fa6e4af9 Sep 10 14:53:28.204785 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:28.204740 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" event={"ID":"04df52ba-2f52-4ea1-9b82-9b562cc362c1","Type":"ContainerStarted","Data":"aa16f69c5dbd546af0287c7aba6819315859b9967ca20736eac34128fa6e4af9"} Sep 10 14:53:31.219831 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:31.219779 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" event={"ID":"04df52ba-2f52-4ea1-9b82-9b562cc362c1","Type":"ContainerStarted","Data":"ce9efe98df8b2284617f2ca5bf4d4271a0d3bf0fca9105795461067d1e8b89c4"} Sep 10 14:53:32.228661 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:32.228624 2560 generic.go:356] "Generic (PLEG): container finished" podID="04df52ba-2f52-4ea1-9b82-9b562cc362c1" containerID="ce9efe98df8b2284617f2ca5bf4d4271a0d3bf0fca9105795461067d1e8b89c4" exitCode=0 Sep 10 14:53:32.229047 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:32.228701 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" event={"ID":"04df52ba-2f52-4ea1-9b82-9b562cc362c1","Type":"ContainerDied","Data":"ce9efe98df8b2284617f2ca5bf4d4271a0d3bf0fca9105795461067d1e8b89c4"} Sep 10 14:53:33.235076 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:33.235044 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" event={"ID":"04df52ba-2f52-4ea1-9b82-9b562cc362c1","Type":"ContainerStarted","Data":"8f3e34c4a412fe227b81bf2a5140aba167d96b36a0eefedcb04baaf7a0317523"} Sep 10 14:53:33.235454 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:33.235133 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:33.254634 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:33.254590 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" podStartSLOduration=2.836542517 podStartE2EDuration="7.254577088s" totalImagesPullingTime="3.421214707s" totalInitContainerRuntime="996.819864ms" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:33.253192412 +0000 UTC m=+352.067775188" watchObservedRunningTime="2026-09-10 14:53:33.254577088 +0000 UTC m=+352.069159839" Sep 10 14:53:44.241877 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:44.241784 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-x6spr" Sep 10 14:53:46.837727 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.837693 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6469457db4-kglvf"] Sep 10 14:53:46.843529 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.843509 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:46.846925 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.846902 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 14:53:46.847059 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.846945 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:53:46.847059 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.846969 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xbw9b\"" Sep 10 14:53:46.847059 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.847027 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 14:53:46.847214 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.847088 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 14:53:46.847252 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.847229 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 14:53:46.859245 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.859221 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6469457db4-kglvf"] Sep 10 14:53:46.944667 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.944638 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-metrics-cert\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:46.944667 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.944671 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-cert\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:46.944923 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.944692 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-manager-config\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:46.944923 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:46.944824 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxgmq\" (UniqueName: \"kubernetes.io/projected/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-kube-api-access-bxgmq\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.045989 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.045962 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-metrics-cert\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.046139 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.045994 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-cert\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.046139 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.046014 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-manager-config\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.046139 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.046057 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxgmq\" (UniqueName: \"kubernetes.io/projected/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-kube-api-access-bxgmq\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.046711 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.046690 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-manager-config\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.048332 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.048310 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-cert\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.048458 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.048439 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-metrics-cert\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.054843 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.054821 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxgmq\" (UniqueName: \"kubernetes.io/projected/1d7cfe96-42a5-421c-b491-9c0d7cc54f59-kube-api-access-bxgmq\") pod \"lws-controller-manager-6469457db4-kglvf\" (UID: \"1d7cfe96-42a5-421c-b491-9c0d7cc54f59\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.153643 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.153617 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:47.425824 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:47.425742 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6469457db4-kglvf"] Sep 10 14:53:47.428936 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:53:47.428907 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7cfe96_42a5_421c_b491_9c0d7cc54f59.slice/crio-f2fad57b776e7eb39aaf867ce72cacc93954223212a32cc82fbccf7f32833e6d WatchSource:0}: Error finding container f2fad57b776e7eb39aaf867ce72cacc93954223212a32cc82fbccf7f32833e6d: Status 404 returned error can't find the container with id f2fad57b776e7eb39aaf867ce72cacc93954223212a32cc82fbccf7f32833e6d Sep 10 14:53:48.304226 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:48.304169 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" event={"ID":"1d7cfe96-42a5-421c-b491-9c0d7cc54f59","Type":"ContainerStarted","Data":"f2fad57b776e7eb39aaf867ce72cacc93954223212a32cc82fbccf7f32833e6d"} Sep 10 14:53:50.314809 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:50.314775 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" event={"ID":"1d7cfe96-42a5-421c-b491-9c0d7cc54f59","Type":"ContainerStarted","Data":"ee632bd500bf60dd38f9d1c6a51b882cef350e62fe14d81d5777ab51b752e412"} Sep 10 14:53:50.315210 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:50.314913 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:53:50.329765 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:50.329724 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" podStartSLOduration=1.908096341 podStartE2EDuration="4.329707462s" totalImagesPullingTime="2.421611121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:50.329132057 +0000 UTC m=+369.143714810" watchObservedRunningTime="2026-09-10 14:53:50.329707462 +0000 UTC m=+369.144290214" Sep 10 14:53:53.380543 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.380507 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf"] Sep 10 14:53:53.383009 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.382988 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.385359 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.385333 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 14:53:53.385359 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.385351 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 14:53:53.385514 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.385401 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-trcg5\"" Sep 10 14:53:53.385514 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.385477 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 14:53:53.396060 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.396040 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf"] Sep 10 14:53:53.504742 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.504712 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzwv\" (UniqueName: \"kubernetes.io/projected/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-kube-api-access-bmzwv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.504931 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.504752 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.504931 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.504774 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.504931 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.504838 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.504931 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.504897 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.505133 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.504939 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.505133 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.504982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.505133 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.505010 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.505133 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.505056 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606118 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606083 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606118 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606122 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606348 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606146 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606348 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606164 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606348 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606193 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606348 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606221 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzwv\" (UniqueName: \"kubernetes.io/projected/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-kube-api-access-bmzwv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606348 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606248 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606348 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606273 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606348 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606303 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606693 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606644 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606750 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606686 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606750 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606715 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.606886 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.606763 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.607259 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.607236 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.608701 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.608674 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.608863 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.608846 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.614709 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.614687 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.615006 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.614981 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzwv\" (UniqueName: \"kubernetes.io/projected/d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4-kube-api-access-bmzwv\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf\" (UID: \"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.693636 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.693582 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:53.962368 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:53.962337 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf"] Sep 10 14:53:53.964731 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:53:53.964701 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09fbb8b_4f13_4542_bae5_7f0a4d7f7ba4.slice/crio-bef8a06af8d79a01c381833fbc882b54b7c62057d5afe29298983ff2b521637d WatchSource:0}: Error finding container bef8a06af8d79a01c381833fbc882b54b7c62057d5afe29298983ff2b521637d: Status 404 returned error can't find the container with id bef8a06af8d79a01c381833fbc882b54b7c62057d5afe29298983ff2b521637d Sep 10 14:53:54.332159 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:54.332085 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" event={"ID":"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4","Type":"ContainerStarted","Data":"bef8a06af8d79a01c381833fbc882b54b7c62057d5afe29298983ff2b521637d"} Sep 10 14:53:56.615068 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:56.615020 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:53:56.615345 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:56.615108 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:53:56.615345 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:56.615137 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:53:57.265101 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.265064 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dtqvt"] Sep 10 14:53:57.267635 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.267616 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:57.269917 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.269890 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qsmb4\"" Sep 10 14:53:57.270013 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.269893 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 14:53:57.277990 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.277969 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dtqvt"] Sep 10 14:53:57.336525 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.336498 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03c2fc40-42f4-4121-836f-2a72f9c96a08-cert\") pod \"odh-model-controller-7998c67dc4-dtqvt\" (UID: \"03c2fc40-42f4-4121-836f-2a72f9c96a08\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:57.336649 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.336533 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tljt\" (UniqueName: \"kubernetes.io/projected/03c2fc40-42f4-4121-836f-2a72f9c96a08-kube-api-access-2tljt\") pod \"odh-model-controller-7998c67dc4-dtqvt\" (UID: \"03c2fc40-42f4-4121-836f-2a72f9c96a08\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:57.348679 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.348650 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" event={"ID":"d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4","Type":"ContainerStarted","Data":"7846c6d5421f06989017d864b25f6988b5dc02dd206eb075099474c1e7e91d0b"} Sep 10 14:53:57.367170 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.367128 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" podStartSLOduration=1.7191028 podStartE2EDuration="4.36711384s" totalImagesPullingTime="2.64801104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:57.365833397 +0000 UTC m=+376.180416146" watchObservedRunningTime="2026-09-10 14:53:57.36711384 +0000 UTC m=+376.181696590" Sep 10 14:53:57.437894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.437551 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03c2fc40-42f4-4121-836f-2a72f9c96a08-cert\") pod \"odh-model-controller-7998c67dc4-dtqvt\" (UID: \"03c2fc40-42f4-4121-836f-2a72f9c96a08\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:57.437894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.437590 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tljt\" (UniqueName: \"kubernetes.io/projected/03c2fc40-42f4-4121-836f-2a72f9c96a08-kube-api-access-2tljt\") pod \"odh-model-controller-7998c67dc4-dtqvt\" (UID: \"03c2fc40-42f4-4121-836f-2a72f9c96a08\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:57.437894 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:53:57.437817 2560 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 14:53:57.438188 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:53:57.437908 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c2fc40-42f4-4121-836f-2a72f9c96a08-cert podName:03c2fc40-42f4-4121-836f-2a72f9c96a08 nodeName:}" failed. No retries permitted until 2026-09-10 14:53:57.937887969 +0000 UTC m=+376.752470702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/03c2fc40-42f4-4121-836f-2a72f9c96a08-cert") pod "odh-model-controller-7998c67dc4-dtqvt" (UID: "03c2fc40-42f4-4121-836f-2a72f9c96a08") : secret "odh-model-controller-webhook-cert" not found Sep 10 14:53:57.446288 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.446263 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tljt\" (UniqueName: \"kubernetes.io/projected/03c2fc40-42f4-4121-836f-2a72f9c96a08-kube-api-access-2tljt\") pod \"odh-model-controller-7998c67dc4-dtqvt\" (UID: \"03c2fc40-42f4-4121-836f-2a72f9c96a08\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:57.694126 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.694094 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:57.698454 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.698431 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:57.926165 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.926127 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v7dgr"] Sep 10 14:53:57.929289 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.929268 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:57.931769 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.931748 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 14:53:57.939229 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.939206 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v7dgr"] Sep 10 14:53:57.941106 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.941082 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03c2fc40-42f4-4121-836f-2a72f9c96a08-cert\") pod \"odh-model-controller-7998c67dc4-dtqvt\" (UID: \"03c2fc40-42f4-4121-836f-2a72f9c96a08\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:57.943641 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:57.943622 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03c2fc40-42f4-4121-836f-2a72f9c96a08-cert\") pod \"odh-model-controller-7998c67dc4-dtqvt\" (UID: \"03c2fc40-42f4-4121-836f-2a72f9c96a08\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:58.042408 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.042342 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v7dgr\" (UID: \"8c96ead6-6b3b-4286-a355-4adfc1e8c78f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:58.042408 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.042372 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtjg\" (UniqueName: \"kubernetes.io/projected/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-kube-api-access-jrtjg\") pod \"model-serving-api-54d89bd5fc-v7dgr\" (UID: \"8c96ead6-6b3b-4286-a355-4adfc1e8c78f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:58.143423 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.143398 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v7dgr\" (UID: \"8c96ead6-6b3b-4286-a355-4adfc1e8c78f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:58.143560 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.143429 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtjg\" (UniqueName: \"kubernetes.io/projected/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-kube-api-access-jrtjg\") pod \"model-serving-api-54d89bd5fc-v7dgr\" (UID: \"8c96ead6-6b3b-4286-a355-4adfc1e8c78f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:58.143600 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:53:58.143561 2560 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 14:53:58.143660 ip-10-0-143-4 kubenswrapper[2560]: E0910 14:53:58.143649 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-tls-certs podName:8c96ead6-6b3b-4286-a355-4adfc1e8c78f nodeName:}" failed. No retries permitted until 2026-09-10 14:53:58.64363163 +0000 UTC m=+377.458214362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-tls-certs") pod "model-serving-api-54d89bd5fc-v7dgr" (UID: "8c96ead6-6b3b-4286-a355-4adfc1e8c78f") : secret "model-serving-api-tls" not found Sep 10 14:53:58.156355 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.156329 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtjg\" (UniqueName: \"kubernetes.io/projected/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-kube-api-access-jrtjg\") pod \"model-serving-api-54d89bd5fc-v7dgr\" (UID: \"8c96ead6-6b3b-4286-a355-4adfc1e8c78f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:58.184501 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.184478 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:53:58.353241 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.353174 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:58.354162 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.354139 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf" Sep 10 14:53:58.454679 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.454562 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dtqvt"] Sep 10 14:53:58.457170 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:53:58.457147 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c2fc40_42f4_4121_836f_2a72f9c96a08.slice/crio-2cddd3817ae790cc13cb6ef6b6130f24651f82834ca6d4b29159d3d1bb219dee WatchSource:0}: Error finding container 2cddd3817ae790cc13cb6ef6b6130f24651f82834ca6d4b29159d3d1bb219dee: Status 404 returned error can't find the container with id 2cddd3817ae790cc13cb6ef6b6130f24651f82834ca6d4b29159d3d1bb219dee Sep 10 14:53:58.649145 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.649058 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v7dgr\" (UID: \"8c96ead6-6b3b-4286-a355-4adfc1e8c78f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:58.651284 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.651259 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c96ead6-6b3b-4286-a355-4adfc1e8c78f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v7dgr\" (UID: \"8c96ead6-6b3b-4286-a355-4adfc1e8c78f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:58.841034 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:58.841011 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:53:59.106979 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:59.106873 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v7dgr"] Sep 10 14:53:59.109522 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:53:59.109492 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c96ead6_6b3b_4286_a355_4adfc1e8c78f.slice/crio-c2be921b515c5e5e21f160662c4d80058309e2a6241a8d5b99bb24ecab6de301 WatchSource:0}: Error finding container c2be921b515c5e5e21f160662c4d80058309e2a6241a8d5b99bb24ecab6de301: Status 404 returned error can't find the container with id c2be921b515c5e5e21f160662c4d80058309e2a6241a8d5b99bb24ecab6de301 Sep 10 14:53:59.361002 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:59.360918 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" event={"ID":"03c2fc40-42f4-4121-836f-2a72f9c96a08","Type":"ContainerStarted","Data":"2cddd3817ae790cc13cb6ef6b6130f24651f82834ca6d4b29159d3d1bb219dee"} Sep 10 14:53:59.362258 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:53:59.362227 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" event={"ID":"8c96ead6-6b3b-4286-a355-4adfc1e8c78f","Type":"ContainerStarted","Data":"c2be921b515c5e5e21f160662c4d80058309e2a6241a8d5b99bb24ecab6de301"} Sep 10 14:54:01.322349 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:01.322321 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6469457db4-kglvf" Sep 10 14:54:03.388333 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.388283 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" event={"ID":"03c2fc40-42f4-4121-836f-2a72f9c96a08","Type":"ContainerStarted","Data":"ad409efdd4452a0a9fe246161a14fcd06c15210bba6a229e3968f94ee9a62ec6"} Sep 10 14:54:03.388863 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.388402 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:54:03.390347 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.390318 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" event={"ID":"8c96ead6-6b3b-4286-a355-4adfc1e8c78f","Type":"ContainerStarted","Data":"801a5d0dc6f891cc0915ac7446755e455bd633ee7db08629202b1016ebb95053"} Sep 10 14:54:03.390550 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.390528 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:54:03.408314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.408255 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" podStartSLOduration=2.537165946 podStartE2EDuration="6.408237311s" totalImagesPullingTime="3.871071365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:54:03.407771011 +0000 UTC m=+382.222353787" watchObservedRunningTime="2026-09-10 14:54:03.408237311 +0000 UTC m=+382.222820063" Sep 10 14:54:03.424024 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.423964 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" podStartSLOduration=3.156395867 podStartE2EDuration="6.423943355s" totalImagesPullingTime="3.267547488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:54:03.423374028 +0000 UTC m=+382.237956780" watchObservedRunningTime="2026-09-10 14:54:03.423943355 +0000 UTC m=+382.238526110" Sep 10 14:54:03.766827 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.766731 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r"] Sep 10 14:54:03.773402 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.773381 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.775817 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.775780 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 14:54:03.775932 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.775815 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 14:54:03.775932 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.775896 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 14:54:03.775932 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.775917 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c7gxb\"" Sep 10 14:54:03.784675 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.784654 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r"] Sep 10 14:54:03.895652 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.895613 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708f4997-9623-4ba9-b6fa-0471141288e7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.895652 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.895648 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.895946 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.895676 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.895946 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.895694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/708f4997-9623-4ba9-b6fa-0471141288e7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.895946 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.895790 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.895946 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.895884 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxpg\" (UniqueName: \"kubernetes.io/projected/708f4997-9623-4ba9-b6fa-0471141288e7-kube-api-access-qvxpg\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.896137 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.895941 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/708f4997-9623-4ba9-b6fa-0471141288e7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.996682 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.996640 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.996894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.996698 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxpg\" (UniqueName: \"kubernetes.io/projected/708f4997-9623-4ba9-b6fa-0471141288e7-kube-api-access-qvxpg\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.996894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.996735 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/708f4997-9623-4ba9-b6fa-0471141288e7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.996894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.996787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708f4997-9623-4ba9-b6fa-0471141288e7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.996894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.996824 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.996894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.996865 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.996894 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.996889 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/708f4997-9623-4ba9-b6fa-0471141288e7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.997669 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.997636 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/708f4997-9623-4ba9-b6fa-0471141288e7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.999040 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.999012 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/708f4997-9623-4ba9-b6fa-0471141288e7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.999232 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.999213 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.999314 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.999253 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:03.999373 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:03.999342 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/708f4997-9623-4ba9-b6fa-0471141288e7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:04.004517 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:04.004498 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708f4997-9623-4ba9-b6fa-0471141288e7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:04.004597 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:04.004573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxpg\" (UniqueName: \"kubernetes.io/projected/708f4997-9623-4ba9-b6fa-0471141288e7-kube-api-access-qvxpg\") pod \"istiod-openshift-gateway-5677ccb49b-j8r5r\" (UID: \"708f4997-9623-4ba9-b6fa-0471141288e7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:04.082604 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:04.082515 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:04.359872 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:04.358825 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r"] Sep 10 14:54:04.367252 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:54:04.367219 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708f4997_9623_4ba9_b6fa_0471141288e7.slice/crio-ed94b1001e78c3c7775fc017ad97a68ead8d1f63ba340c908c00f8035cdb2fb5 WatchSource:0}: Error finding container ed94b1001e78c3c7775fc017ad97a68ead8d1f63ba340c908c00f8035cdb2fb5: Status 404 returned error can't find the container with id ed94b1001e78c3c7775fc017ad97a68ead8d1f63ba340c908c00f8035cdb2fb5 Sep 10 14:54:04.396198 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:04.396172 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" event={"ID":"708f4997-9623-4ba9-b6fa-0471141288e7","Type":"ContainerStarted","Data":"ed94b1001e78c3c7775fc017ad97a68ead8d1f63ba340c908c00f8035cdb2fb5"} Sep 10 14:54:07.041656 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:07.041621 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:54:07.041945 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:07.041688 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:54:07.417023 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:07.416963 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" event={"ID":"708f4997-9623-4ba9-b6fa-0471141288e7","Type":"ContainerStarted","Data":"5df9e2c1c328b09b305a07d6a7f0a1f569a6d3062a3191b6810156567dd9240d"} Sep 10 14:54:07.417230 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:07.417108 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:07.440975 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:07.440897 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" podStartSLOduration=1.773706984 podStartE2EDuration="4.440876808s" totalImagesPullingTime="2.667169824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:54:07.43967203 +0000 UTC m=+386.254254800" watchObservedRunningTime="2026-09-10 14:54:07.440876808 +0000 UTC m=+386.255459578" Sep 10 14:54:08.422553 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:08.422521 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8r5r" Sep 10 14:54:14.399105 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:14.399076 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-dtqvt" Sep 10 14:54:14.402712 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:54:14.402690 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-v7dgr" Sep 10 14:56:24.842945 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.842910 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb"] Sep 10 14:56:24.845717 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.845697 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.848934 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.848910 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hgm28\"" Sep 10 14:56:24.858031 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.858006 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb"] Sep 10 14:56:24.896642 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896616 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.896819 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896652 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.896819 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896672 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.896819 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896692 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.896819 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896747 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.896819 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896781 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c1404d-585b-46bf-98b8-7ca1985a600d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.897017 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896864 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.897017 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896881 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9pk\" (UniqueName: \"kubernetes.io/projected/39c1404d-585b-46bf-98b8-7ca1985a600d-kube-api-access-cq9pk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.897017 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.896907 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998236 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998199 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998243 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9pk\" (UniqueName: \"kubernetes.io/projected/39c1404d-585b-46bf-98b8-7ca1985a600d-kube-api-access-cq9pk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998276 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998336 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998373 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998401 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998694 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998432 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998694 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998694 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998501 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c1404d-585b-46bf-98b8-7ca1985a600d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998880 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998698 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998937 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998896 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.998998 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.998980 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.999055 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.999008 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:24.999233 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:24.999213 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c1404d-585b-46bf-98b8-7ca1985a600d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:25.000638 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.000613 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:25.000889 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.000872 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:25.009166 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.009145 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c1404d-585b-46bf-98b8-7ca1985a600d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:25.009369 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.009348 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9pk\" (UniqueName: \"kubernetes.io/projected/39c1404d-585b-46bf-98b8-7ca1985a600d-kube-api-access-cq9pk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-t22tb\" (UID: \"39c1404d-585b-46bf-98b8-7ca1985a600d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:25.159370 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.159332 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:25.434369 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.434315 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb"] Sep 10 14:56:25.436614 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:56:25.436591 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c1404d_585b_46bf_98b8_7ca1985a600d.slice/crio-beb0382629357609775a82cd614ff4098bcfed72dbc797dae1b8223b3eab14ba WatchSource:0}: Error finding container beb0382629357609775a82cd614ff4098bcfed72dbc797dae1b8223b3eab14ba: Status 404 returned error can't find the container with id beb0382629357609775a82cd614ff4098bcfed72dbc797dae1b8223b3eab14ba Sep 10 14:56:25.438659 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.438625 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:56:25.438727 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.438686 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:56:25.438727 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:25.438715 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:56:26.033056 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:26.033020 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" event={"ID":"39c1404d-585b-46bf-98b8-7ca1985a600d","Type":"ContainerStarted","Data":"b25e74cdff5d38d20e2f14a149b75f1e62543340e3afdf4d063964b3630f938f"} Sep 10 14:56:26.033426 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:26.033071 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" event={"ID":"39c1404d-585b-46bf-98b8-7ca1985a600d","Type":"ContainerStarted","Data":"beb0382629357609775a82cd614ff4098bcfed72dbc797dae1b8223b3eab14ba"} Sep 10 14:56:26.053674 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:26.053627 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" podStartSLOduration=2.053612629 podStartE2EDuration="2.053612629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:26.052374581 +0000 UTC m=+524.866957333" watchObservedRunningTime="2026-09-10 14:56:26.053612629 +0000 UTC m=+524.868195381" Sep 10 14:56:26.159967 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:26.159936 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:26.169561 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:26.169534 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:27.038162 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:27.038130 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:56:27.038945 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:56:27.038926 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:57:41.613820 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:41.613770 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:57:41.614315 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:41.613829 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 14:57:49.817688 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.817649 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt"] Sep 10 14:57:49.821171 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.821154 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:49.824023 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.823997 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 14:57:49.824155 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.824137 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:57:49.824225 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.824116 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 14:57:49.825100 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.825078 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vh2ph\"" Sep 10 14:57:49.825480 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.825457 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 14:57:49.838770 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.838747 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt"] Sep 10 14:57:49.917626 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.917597 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-metrics-certs\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:49.917763 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.917657 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-openshift-service-ca\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:49.917830 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.917768 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-maas-api-tls\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:49.917830 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:49.917817 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcw8\" (UniqueName: \"kubernetes.io/projected/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-kube-api-access-wzcw8\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.018699 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.018673 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-openshift-service-ca\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.018865 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.018724 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-maas-api-tls\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.018865 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.018828 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcw8\" (UniqueName: \"kubernetes.io/projected/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-kube-api-access-wzcw8\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.018982 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.018894 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-metrics-certs\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.019360 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.019337 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-openshift-service-ca\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.020943 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.020924 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-maas-api-tls\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.021020 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.021009 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-metrics-certs\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.027052 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.027035 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcw8\" (UniqueName: \"kubernetes.io/projected/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-kube-api-access-wzcw8\") pod \"maas-api-9d7854d48-gxmkt\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.136724 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.136700 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:50.424960 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:50.424937 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt"] Sep 10 14:57:50.427355 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:57:50.427323 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0f3cc0_ff2f_40ee_970f_baa8a11eac20.slice/crio-45f542690eec7ac6a927bead6229ab166407caba99aca887b16b28639a3bbba2 WatchSource:0}: Error finding container 45f542690eec7ac6a927bead6229ab166407caba99aca887b16b28639a3bbba2: Status 404 returned error can't find the container with id 45f542690eec7ac6a927bead6229ab166407caba99aca887b16b28639a3bbba2 Sep 10 14:57:51.421903 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:51.421862 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" event={"ID":"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20","Type":"ContainerStarted","Data":"45f542690eec7ac6a927bead6229ab166407caba99aca887b16b28639a3bbba2"} Sep 10 14:57:53.433783 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:53.433751 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" event={"ID":"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20","Type":"ContainerStarted","Data":"23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708"} Sep 10 14:57:53.434141 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:53.433880 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:57:53.454139 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:53.454084 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" podStartSLOduration=1.64314086 podStartE2EDuration="4.454063867s" totalImagesPullingTime="2.810923007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:53.451535107 +0000 UTC m=+612.266117858" watchObservedRunningTime="2026-09-10 14:57:53.454063867 +0000 UTC m=+612.268646618" Sep 10 14:57:58.079976 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:58.079934 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-t22tb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 14:57:58.080461 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:58.080012 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" podUID="39c1404d-585b-46bf-98b8-7ca1985a600d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 14:57:59.443585 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:57:59.443557 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 14:58:00.467031 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:00.467003 2560 generic.go:356] "Generic (PLEG): container finished" podID="39c1404d-585b-46bf-98b8-7ca1985a600d" containerID="b25e74cdff5d38d20e2f14a149b75f1e62543340e3afdf4d063964b3630f938f" exitCode=0 Sep 10 14:58:00.467520 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:00.467081 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" event={"ID":"39c1404d-585b-46bf-98b8-7ca1985a600d","Type":"ContainerDied","Data":"b25e74cdff5d38d20e2f14a149b75f1e62543340e3afdf4d063964b3630f938f"} Sep 10 14:58:00.467584 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:00.467569 2560 scope.go:117] "RemoveContainer" containerID="b25e74cdff5d38d20e2f14a149b75f1e62543340e3afdf4d063964b3630f938f" Sep 10 14:58:00.468512 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:00.468486 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:58:00.468605 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:00.468567 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:58:00.468672 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:00.468616 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:58:01.476913 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:01.476876 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" event={"ID":"39c1404d-585b-46bf-98b8-7ca1985a600d","Type":"ContainerStarted","Data":"68f096638089cc7d2204c6202a27d588b17f8eb2d973b81496d875a1dac89e2d"} Sep 10 14:58:02.160066 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:02.160028 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:58:02.160460 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:02.160432 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-t22tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 14:58:02.160559 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:02.160508 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" podUID="39c1404d-585b-46bf-98b8-7ca1985a600d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 14:58:03.160252 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:03.160207 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-t22tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 14:58:03.160757 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:03.160281 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" podUID="39c1404d-585b-46bf-98b8-7ca1985a600d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 14:58:04.160356 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:04.160253 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-t22tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 14:58:04.160356 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:04.160325 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" podUID="39c1404d-585b-46bf-98b8-7ca1985a600d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 14:58:05.162163 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:05.162126 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-t22tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 14:58:05.162719 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:05.162685 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" podUID="39c1404d-585b-46bf-98b8-7ca1985a600d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 14:58:06.160032 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:06.159996 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-t22tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 14:58:06.161130 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:06.161095 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" podUID="39c1404d-585b-46bf-98b8-7ca1985a600d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 14:58:07.163130 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:07.163099 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:58:07.163558 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:07.163349 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:58:07.163899 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:07.163881 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-t22tb" Sep 10 14:58:24.150722 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.150688 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds"] Sep 10 14:58:24.154178 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.154160 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.157180 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.157163 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 14:58:24.157503 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.157489 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 14:58:24.158317 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.158300 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5vv4r\"" Sep 10 14:58:24.158424 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.158405 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 14:58:24.166749 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.166730 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds"] Sep 10 14:58:24.231816 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.231770 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.231967 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.231825 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa1b823d-26d0-444d-9eb0-8f960673787e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.231967 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.231888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hscxz\" (UniqueName: \"kubernetes.io/projected/fa1b823d-26d0-444d-9eb0-8f960673787e-kube-api-access-hscxz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.231967 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.231959 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.232132 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.232011 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.232132 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.232052 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.332927 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.332892 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hscxz\" (UniqueName: \"kubernetes.io/projected/fa1b823d-26d0-444d-9eb0-8f960673787e-kube-api-access-hscxz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.333073 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.332956 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.333073 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.332999 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.333073 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.333049 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.333252 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.333077 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.333252 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.333105 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa1b823d-26d0-444d-9eb0-8f960673787e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.333414 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.333382 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.333414 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.333407 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.333550 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.333532 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.335353 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.335333 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa1b823d-26d0-444d-9eb0-8f960673787e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.335517 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.335501 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa1b823d-26d0-444d-9eb0-8f960673787e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.344454 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.344413 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hscxz\" (UniqueName: \"kubernetes.io/projected/fa1b823d-26d0-444d-9eb0-8f960673787e-kube-api-access-hscxz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rcbds\" (UID: \"fa1b823d-26d0-444d-9eb0-8f960673787e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.465076 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.465009 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:24.741264 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.741237 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds"] Sep 10 14:58:24.742736 ip-10-0-143-4 kubenswrapper[2560]: W0910 14:58:24.742708 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1b823d_26d0_444d_9eb0_8f960673787e.slice/crio-86432a74dc682a761e728165a39246a3bc268a2ac3f2c3bd8d013fe0fde95531 WatchSource:0}: Error finding container 86432a74dc682a761e728165a39246a3bc268a2ac3f2c3bd8d013fe0fde95531: Status 404 returned error can't find the container with id 86432a74dc682a761e728165a39246a3bc268a2ac3f2c3bd8d013fe0fde95531 Sep 10 14:58:24.744069 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:24.744054 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:58:25.590185 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:25.590151 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" event={"ID":"fa1b823d-26d0-444d-9eb0-8f960673787e","Type":"ContainerStarted","Data":"86432a74dc682a761e728165a39246a3bc268a2ac3f2c3bd8d013fe0fde95531"} Sep 10 14:58:26.596640 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:26.596606 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" event={"ID":"fa1b823d-26d0-444d-9eb0-8f960673787e","Type":"ContainerStarted","Data":"8b606376e01a6eb31b0bb3066b7cb3bdf1780a3a7e9894c25a6ab4dfc163d334"} Sep 10 14:58:26.617729 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:26.617686 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" podStartSLOduration=1.5823394560000001 podStartE2EDuration="2.617672595s" totalImagesPullingTime="1.035333139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:26.616365643 +0000 UTC m=+645.430948396" watchObservedRunningTime="2026-09-10 14:58:26.617672595 +0000 UTC m=+645.432255346" Sep 10 14:58:34.465611 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:34.465575 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:34.466006 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:34.465650 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:34.477790 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:34.477768 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 14:58:34.648182 ip-10-0-143-4 kubenswrapper[2560]: I0910 14:58:34.648154 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rcbds" Sep 10 15:01:25.761171 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.761140 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb"] Sep 10 15:01:25.765194 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.765172 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.768078 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.768057 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e577f0-openshift-default-dockercfg-cmxx7\"" Sep 10 15:01:25.792705 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.792677 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb"] Sep 10 15:01:25.924644 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-credential-socket\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.924855 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924657 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4a061b-907b-4597-a054-c8d89dffe728-istio-podinfo\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.924855 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924715 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4a061b-907b-4597-a054-c8d89dffe728-istiod-ca-cert\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.924855 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924749 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-envoy\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.924855 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924771 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bk2f\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-kube-api-access-6bk2f\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.924855 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924807 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-socket\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.924855 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924831 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-data\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.925133 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924896 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-certs\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:25.925133 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:25.924948 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-istio-token\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026058 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.025985 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-envoy\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026058 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026016 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bk2f\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-kube-api-access-6bk2f\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026058 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026038 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-socket\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026308 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026063 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-data\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026308 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026090 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-certs\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026308 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026118 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-istio-token\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026308 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026234 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-credential-socket\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026308 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026276 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4a061b-907b-4597-a054-c8d89dffe728-istio-podinfo\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026315 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4a061b-907b-4597-a054-c8d89dffe728-istiod-ca-cert\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026486 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-socket\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026659 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026569 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-data\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026659 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026635 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-certs\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026738 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026673 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-credential-socket\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.026984 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.026964 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4a061b-907b-4597-a054-c8d89dffe728-istiod-ca-cert\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.028228 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.028212 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-envoy\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.028763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.028746 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4a061b-907b-4597-a054-c8d89dffe728-istio-podinfo\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.035452 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.035430 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bk2f\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-kube-api-access-6bk2f\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.036240 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.036222 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-istio-token\") pod \"e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.078098 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.078079 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:26.352869 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.352847 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb"] Sep 10 15:01:26.355021 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:01:26.354992 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4a061b_907b_4597_a054_c8d89dffe728.slice/crio-db0b0e9955de4c3bdad7def35e9c030dc0ac61cee25960944e6dc764d1824988 WatchSource:0}: Error finding container db0b0e9955de4c3bdad7def35e9c030dc0ac61cee25960944e6dc764d1824988: Status 404 returned error can't find the container with id db0b0e9955de4c3bdad7def35e9c030dc0ac61cee25960944e6dc764d1824988 Sep 10 15:01:26.357149 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.357116 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:26.357222 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.357182 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:26.357222 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.357210 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:26.439065 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.439032 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" event={"ID":"be4a061b-907b-4597-a054-c8d89dffe728","Type":"ContainerStarted","Data":"cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23"} Sep 10 15:01:26.439238 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.439089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" event={"ID":"be4a061b-907b-4597-a054-c8d89dffe728","Type":"ContainerStarted","Data":"db0b0e9955de4c3bdad7def35e9c030dc0ac61cee25960944e6dc764d1824988"} Sep 10 15:01:26.463050 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:26.462999 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podStartSLOduration=1.462984354 podStartE2EDuration="1.462984354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:26.462567566 +0000 UTC m=+825.277150320" watchObservedRunningTime="2026-09-10 15:01:26.462984354 +0000 UTC m=+825.277567105" Sep 10 15:01:27.078865 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:27.078838 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:27.083405 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:27.083376 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:27.444568 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:27.444537 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:27.445494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:27.445477 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:29.855147 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:29.855114 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr"] Sep 10 15:01:29.858872 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:29.858854 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:29.864837 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:29.864790 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e577f0-serving-cert\"" Sep 10 15:01:29.881359 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:29.881332 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr"] Sep 10 15:01:29.964399 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:29.964363 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:29.964556 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:29.964487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8466e73-ba2c-474b-892b-a2ca11bb9fad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:29.964556 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:29.964539 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-metrics-certs\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:29.964629 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:29.964556 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndnv\" (UniqueName: \"kubernetes.io/projected/f8466e73-ba2c-474b-892b-a2ca11bb9fad-kube-api-access-zndnv\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.065955 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.065914 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8466e73-ba2c-474b-892b-a2ca11bb9fad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.066130 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.065967 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-metrics-certs\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.066130 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.066073 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndnv\" (UniqueName: \"kubernetes.io/projected/f8466e73-ba2c-474b-892b-a2ca11bb9fad-kube-api-access-zndnv\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.066130 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.066125 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.066276 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:01:30.066259 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-serving-cert: secret "maas-api-e2e-inf-a-e577f0-serving-cert" not found Sep 10 15:01:30.066356 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:01:30.066344 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls podName:f8466e73-ba2c-474b-892b-a2ca11bb9fad nodeName:}" failed. No retries permitted until 2026-09-10 15:01:30.566324294 +0000 UTC m=+829.380907039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls") pod "maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" (UID: "f8466e73-ba2c-474b-892b-a2ca11bb9fad") : secret "maas-api-e2e-inf-a-e577f0-serving-cert" not found Sep 10 15:01:30.066726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.066695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8466e73-ba2c-474b-892b-a2ca11bb9fad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.068508 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.068486 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-metrics-certs\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.075858 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.075834 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndnv\" (UniqueName: \"kubernetes.io/projected/f8466e73-ba2c-474b-892b-a2ca11bb9fad-kube-api-access-zndnv\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.572330 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.572291 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.575087 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.575051 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls\") pod \"maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:30.771102 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:30.771059 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:31.061503 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:31.061470 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr"] Sep 10 15:01:31.064724 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:01:31.064695 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8466e73_ba2c_474b_892b_a2ca11bb9fad.slice/crio-948282ccbb8d017ece9d4f2ff424d9ddc43e5aacfc8dba57bcf75f9910e53522 WatchSource:0}: Error finding container 948282ccbb8d017ece9d4f2ff424d9ddc43e5aacfc8dba57bcf75f9910e53522: Status 404 returned error can't find the container with id 948282ccbb8d017ece9d4f2ff424d9ddc43e5aacfc8dba57bcf75f9910e53522 Sep 10 15:01:31.467062 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:31.467027 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" event={"ID":"f8466e73-ba2c-474b-892b-a2ca11bb9fad","Type":"ContainerStarted","Data":"948282ccbb8d017ece9d4f2ff424d9ddc43e5aacfc8dba57bcf75f9910e53522"} Sep 10 15:01:32.473963 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:32.473929 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" event={"ID":"f8466e73-ba2c-474b-892b-a2ca11bb9fad","Type":"ContainerStarted","Data":"d6090e1121ba90e8d840bb07cca8a720ec1aa7ec7edcea2312375c8342080729"} Sep 10 15:01:32.474326 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:32.473992 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:34.052601 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.052541 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" podStartSLOduration=4.507101616 podStartE2EDuration="5.052524965s" totalImagesPullingTime="545.423349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:32.493264178 +0000 UTC m=+831.307846930" watchObservedRunningTime="2026-09-10 15:01:34.052524965 +0000 UTC m=+832.867107729" Sep 10 15:01:34.053129 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.053108 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd"] Sep 10 15:01:34.056903 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.056881 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.060868 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.060791 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-24e113dd-openshift-default-dockercfg-7cw9c\"" Sep 10 15:01:34.071363 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.071338 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd"] Sep 10 15:01:34.205565 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205530 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-envoy\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.205565 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205564 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd23311-61e5-4d82-b151-11f175272b52-istiod-ca-cert\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.205763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205586 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-socket\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.205763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205690 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-data\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.205763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205735 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-certs\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.205763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205754 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzqv\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-kube-api-access-jrzqv\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.205926 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-istio-token\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.205926 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-credential-socket\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.205926 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.205918 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd23311-61e5-4d82-b151-11f175272b52-istio-podinfo\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.306901 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.306824 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-certs\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.306901 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.306872 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzqv\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-kube-api-access-jrzqv\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307064 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.306903 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-istio-token\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307064 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.306994 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-credential-socket\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307064 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307039 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd23311-61e5-4d82-b151-11f175272b52-istio-podinfo\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307421 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307088 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-envoy\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307421 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307112 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd23311-61e5-4d82-b151-11f175272b52-istiod-ca-cert\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307421 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307142 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-socket\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307421 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-data\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307421 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307275 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-certs\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307421 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307403 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-credential-socket\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307760 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307490 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-data\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307760 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-socket\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.307760 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.307678 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd23311-61e5-4d82-b151-11f175272b52-istiod-ca-cert\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.309986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.309968 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-envoy\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.310215 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.310196 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd23311-61e5-4d82-b151-11f175272b52-istio-podinfo\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.315248 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.315225 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzqv\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-kube-api-access-jrzqv\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.315849 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.315832 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-istio-token\") pod \"e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.368181 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.368151 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:34.644952 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.644784 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd"] Sep 10 15:01:34.647242 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:01:34.647212 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd23311_61e5_4d82_b151_11f175272b52.slice/crio-70acd1ca69eb680283c44412a282361efe2e2aaa5d497f9a6deecc4b3c43beae WatchSource:0}: Error finding container 70acd1ca69eb680283c44412a282361efe2e2aaa5d497f9a6deecc4b3c43beae: Status 404 returned error can't find the container with id 70acd1ca69eb680283c44412a282361efe2e2aaa5d497f9a6deecc4b3c43beae Sep 10 15:01:34.652556 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.650621 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:34.652556 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.650700 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:34.652556 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:34.650739 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:35.488860 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:35.488825 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" event={"ID":"0cd23311-61e5-4d82-b151-11f175272b52","Type":"ContainerStarted","Data":"1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150"} Sep 10 15:01:35.489230 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:35.488876 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" event={"ID":"0cd23311-61e5-4d82-b151-11f175272b52","Type":"ContainerStarted","Data":"70acd1ca69eb680283c44412a282361efe2e2aaa5d497f9a6deecc4b3c43beae"} Sep 10 15:01:35.512139 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:35.512092 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" podStartSLOduration=1.512078507 podStartE2EDuration="1.512078507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:35.510141004 +0000 UTC m=+834.324723752" watchObservedRunningTime="2026-09-10 15:01:35.512078507 +0000 UTC m=+834.326661258" Sep 10 15:01:36.368567 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.368528 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:36.370785 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.370756 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx"] Sep 10 15:01:36.375129 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.375107 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:36.375233 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.375153 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.377515 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.377496 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c75e68-openshift-default-dockercfg-cp7sk\"" Sep 10 15:01:36.388079 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.388055 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx"] Sep 10 15:01:36.496049 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.496018 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:36.497318 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.497296 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:36.529833 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.529779 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-certs\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.529994 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.529883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb72069-209b-4ce9-bdd9-55f44f034466-istiod-ca-cert\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.530079 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.529991 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-credential-socket\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.530079 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.530041 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-socket\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.530192 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.530083 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-data\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.530192 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.530180 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbwrz\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-kube-api-access-zbwrz\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.530309 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.530214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-istio-token\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.530309 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.530268 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-envoy\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.530405 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.530313 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb72069-209b-4ce9-bdd9-55f44f034466-istio-podinfo\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.630980 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.630911 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb72069-209b-4ce9-bdd9-55f44f034466-istio-podinfo\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.630980 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.630952 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-certs\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631163 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631092 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb72069-209b-4ce9-bdd9-55f44f034466-istiod-ca-cert\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631207 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631196 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-credential-socket\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631250 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631233 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-socket\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631444 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631421 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-data\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631561 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631538 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbwrz\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-kube-api-access-zbwrz\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631638 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631574 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-credential-socket\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631638 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631600 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-certs\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631638 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631576 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-istio-token\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631892 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631676 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-envoy\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.631952 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631927 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-data\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.632001 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.631986 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb72069-209b-4ce9-bdd9-55f44f034466-istiod-ca-cert\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.632054 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.632037 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-socket\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.633442 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.633423 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb72069-209b-4ce9-bdd9-55f44f034466-istio-podinfo\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.633667 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.633653 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-envoy\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.639468 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.639445 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-istio-token\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.639760 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.639741 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbwrz\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-kube-api-access-zbwrz\") pod \"e2e-autores-c75e68-openshift-default-59d978c98-ckkfx\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.689122 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.689055 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:36.977813 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.977705 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx"] Sep 10 15:01:36.983412 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:01:36.983378 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb72069_209b_4ce9_bdd9_55f44f034466.slice/crio-b919780dc2ae82ab46605789369ed90669bb28fb20b17109bb593e5950267b19 WatchSource:0}: Error finding container b919780dc2ae82ab46605789369ed90669bb28fb20b17109bb593e5950267b19: Status 404 returned error can't find the container with id b919780dc2ae82ab46605789369ed90669bb28fb20b17109bb593e5950267b19 Sep 10 15:01:36.986150 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.986110 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:36.986249 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.986201 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:36.986322 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:36.986247 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:37.504164 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:37.504128 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" event={"ID":"abb72069-209b-4ce9-bdd9-55f44f034466","Type":"ContainerStarted","Data":"68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df"} Sep 10 15:01:37.504637 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:37.504217 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" event={"ID":"abb72069-209b-4ce9-bdd9-55f44f034466","Type":"ContainerStarted","Data":"b919780dc2ae82ab46605789369ed90669bb28fb20b17109bb593e5950267b19"} Sep 10 15:01:37.527926 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:37.527877 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" podStartSLOduration=1.527862791 podStartE2EDuration="1.527862791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:37.526229034 +0000 UTC m=+836.340811844" watchObservedRunningTime="2026-09-10 15:01:37.527862791 +0000 UTC m=+836.342445541" Sep 10 15:01:37.689880 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:37.689841 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:37.967304 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:37.967273 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k"] Sep 10 15:01:37.976283 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:37.976260 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:37.981191 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:37.981169 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-24e113dd-serving-cert\"" Sep 10 15:01:37.981559 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:37.981347 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k"] Sep 10 15:01:38.149471 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.149435 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.149666 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.149495 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-metrics-certs\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.149666 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.149515 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2hk\" (UniqueName: \"kubernetes.io/projected/5c46aa96-e197-40c6-bc4a-3599571d86b3-kube-api-access-9k2hk\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.149666 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.149578 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c46aa96-e197-40c6-bc4a-3599571d86b3-openshift-service-ca\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.250157 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.250070 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.250157 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.250128 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-metrics-certs\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.250157 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.250145 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2hk\" (UniqueName: \"kubernetes.io/projected/5c46aa96-e197-40c6-bc4a-3599571d86b3-kube-api-access-9k2hk\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.250393 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.250194 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c46aa96-e197-40c6-bc4a-3599571d86b3-openshift-service-ca\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.250393 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:01:38.250223 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-serving-cert: secret "maas-api-e2e-ait-24e113dd-serving-cert" not found Sep 10 15:01:38.250393 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:01:38.250295 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls podName:5c46aa96-e197-40c6-bc4a-3599571d86b3 nodeName:}" failed. No retries permitted until 2026-09-10 15:01:38.750275268 +0000 UTC m=+837.564858010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls") pod "maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" (UID: "5c46aa96-e197-40c6-bc4a-3599571d86b3") : secret "maas-api-e2e-ait-24e113dd-serving-cert" not found Sep 10 15:01:38.250913 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.250890 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c46aa96-e197-40c6-bc4a-3599571d86b3-openshift-service-ca\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.252964 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.252941 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-metrics-certs\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.259405 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.259383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2hk\" (UniqueName: \"kubernetes.io/projected/5c46aa96-e197-40c6-bc4a-3599571d86b3-kube-api-access-9k2hk\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.488790 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.488762 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:01:38.694373 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.694343 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:38.772821 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.772202 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.776823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.775895 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls\") pod \"maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:38.912867 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:38.912832 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:39.208636 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:39.208612 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k"] Sep 10 15:01:39.210431 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:01:39.210403 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c46aa96_e197_40c6_bc4a_3599571d86b3.slice/crio-b5d52decc29ee54c2b0d11cc42b3a96ec0b6c43cd13cb61ca3a3c4e438b66669 WatchSource:0}: Error finding container b5d52decc29ee54c2b0d11cc42b3a96ec0b6c43cd13cb61ca3a3c4e438b66669: Status 404 returned error can't find the container with id b5d52decc29ee54c2b0d11cc42b3a96ec0b6c43cd13cb61ca3a3c4e438b66669 Sep 10 15:01:39.516624 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:39.516591 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" event={"ID":"5c46aa96-e197-40c6-bc4a-3599571d86b3","Type":"ContainerStarted","Data":"b5d52decc29ee54c2b0d11cc42b3a96ec0b6c43cd13cb61ca3a3c4e438b66669"} Sep 10 15:01:39.516937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:39.516915 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:39.517878 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:39.517850 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:40.526983 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:40.526950 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" event={"ID":"5c46aa96-e197-40c6-bc4a-3599571d86b3","Type":"ContainerStarted","Data":"4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b"} Sep 10 15:01:40.527411 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:40.527152 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:40.546253 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:40.546204 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" podStartSLOduration=2.771969171 podStartE2EDuration="3.546189913s" totalImagesPullingTime="774.220742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:40.545234591 +0000 UTC m=+839.359817357" watchObservedRunningTime="2026-09-10 15:01:40.546189913 +0000 UTC m=+839.360772665" Sep 10 15:01:43.485927 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:43.485893 2560 patch_prober.go:28] interesting pod/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:01:43.486376 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:43.485943 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:01:45.562876 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:45.562841 2560 generic.go:356] "Generic (PLEG): container finished" podID="be4a061b-907b-4597-a054-c8d89dffe728" containerID="cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23" exitCode=0 Sep 10 15:01:45.563284 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:45.562895 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" event={"ID":"be4a061b-907b-4597-a054-c8d89dffe728","Type":"ContainerDied","Data":"cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23"} Sep 10 15:01:45.563328 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:45.563288 2560 scope.go:117] "RemoveContainer" containerID="cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23" Sep 10 15:01:45.564334 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:45.564305 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:45.564415 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:45.564402 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:45.564453 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:45.564435 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:46.538240 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:46.538214 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:01:46.580967 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:46.580926 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" event={"ID":"be4a061b-907b-4597-a054-c8d89dffe728","Type":"ContainerStarted","Data":"b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d"} Sep 10 15:01:47.079175 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:47.079138 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:47.079610 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:47.079523 2560 patch_prober.go:28] interesting pod/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 10 15:01:47.079819 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:47.079773 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 10 15:01:48.078682 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:48.078644 2560 patch_prober.go:28] interesting pod/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 10 15:01:48.079318 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:48.079285 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 10 15:01:49.078588 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:49.078542 2560 patch_prober.go:28] interesting pod/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 10 15:01:49.078829 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:49.078625 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 10 15:01:50.079032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:50.078989 2560 patch_prober.go:28] interesting pod/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 10 15:01:50.079538 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:50.079058 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 10 15:01:51.082730 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:51.082289 2560 patch_prober.go:28] interesting pod/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 10 15:01:51.082730 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:51.082352 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 10 15:01:52.087318 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:52.087286 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:52.089158 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:52.089136 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:52.095309 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:52.095290 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:01:54.121870 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.121837 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp"] Sep 10 15:01:54.124994 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.124961 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.129145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.129121 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6356e4-openshift-default-dockercfg-76swn\"" Sep 10 15:01:54.148248 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.148220 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp"] Sep 10 15:01:54.251444 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251412 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-envoy\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.251444 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251449 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-token\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.251690 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251480 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-certs\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.251690 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251584 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-data\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.251690 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251605 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc0e1a4-20b3-4144-b4e1-382919c300da-istiod-ca-cert\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.251690 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251659 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-socket\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.251947 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251709 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjt6\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-kube-api-access-2zjt6\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.251947 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251762 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-podinfo\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.251947 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.251842 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-credential-socket\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.352907 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.352869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-podinfo\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353097 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.352930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-credential-socket\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353097 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-envoy\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353097 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-token\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353345 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-certs\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353345 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-data\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353345 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353206 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc0e1a4-20b3-4144-b4e1-382919c300da-istiod-ca-cert\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353345 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353262 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-socket\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353345 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353311 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjt6\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-kube-api-access-2zjt6\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353732 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353704 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-certs\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.353934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.353913 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-data\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.354035 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.354013 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-credential-socket\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.355021 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.354538 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc0e1a4-20b3-4144-b4e1-382919c300da-istiod-ca-cert\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.355021 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.354761 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-socket\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.357088 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.357065 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-podinfo\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.357594 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.357558 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-envoy\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.368004 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.367983 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjt6\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-kube-api-access-2zjt6\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.368461 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.368441 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-token\") pod \"e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.438252 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.438172 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:54.735278 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.735253 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp"] Sep 10 15:01:54.737959 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:01:54.737935 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc0e1a4_20b3_4144_b4e1_382919c300da.slice/crio-76534be8ba9bd8f062e7ab2af4bb87431ae01035bbfdff1031b9cd5bcfee4371 WatchSource:0}: Error finding container 76534be8ba9bd8f062e7ab2af4bb87431ae01035bbfdff1031b9cd5bcfee4371: Status 404 returned error can't find the container with id 76534be8ba9bd8f062e7ab2af4bb87431ae01035bbfdff1031b9cd5bcfee4371 Sep 10 15:01:54.740976 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.740936 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:54.741068 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.741019 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:54.741068 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:54.741059 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:55.373775 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.373467 2560 patch_prober.go:28] interesting pod/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": read tcp 10.133.0.2:50348->10.133.0.35:15021: read: connection reset by peer" start-of-body= Sep 10 15:01:55.373775 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.373566 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": read tcp 10.133.0.2:50348->10.133.0.35:15021: read: connection reset by peer" Sep 10 15:01:55.642310 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.642238 2560 generic.go:356] "Generic (PLEG): container finished" podID="abb72069-209b-4ce9-bdd9-55f44f034466" containerID="68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df" exitCode=0 Sep 10 15:01:55.642467 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.642320 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" event={"ID":"abb72069-209b-4ce9-bdd9-55f44f034466","Type":"ContainerDied","Data":"68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df"} Sep 10 15:01:55.642940 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.642916 2560 scope.go:117] "RemoveContainer" containerID="68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df" Sep 10 15:01:55.644103 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.644081 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" event={"ID":"0dc0e1a4-20b3-4144-b4e1-382919c300da","Type":"ContainerStarted","Data":"78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec"} Sep 10 15:01:55.644229 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.644135 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" event={"ID":"0dc0e1a4-20b3-4144-b4e1-382919c300da","Type":"ContainerStarted","Data":"76534be8ba9bd8f062e7ab2af4bb87431ae01035bbfdff1031b9cd5bcfee4371"} Sep 10 15:01:55.644298 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.644239 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:55.644355 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.644332 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:55.644410 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.644380 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:55.706654 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:55.706605 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" podStartSLOduration=1.7065873630000001 podStartE2EDuration="1.706587363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:55.705270309 +0000 UTC m=+854.519853062" watchObservedRunningTime="2026-09-10 15:01:55.706587363 +0000 UTC m=+854.521170105" Sep 10 15:01:56.438332 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.438299 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:56.443309 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.443286 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:56.651197 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.651157 2560 generic.go:356] "Generic (PLEG): container finished" podID="0cd23311-61e5-4d82-b151-11f175272b52" containerID="1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150" exitCode=0 Sep 10 15:01:56.651383 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.651229 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" event={"ID":"0cd23311-61e5-4d82-b151-11f175272b52","Type":"ContainerDied","Data":"1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150"} Sep 10 15:01:56.651706 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.651689 2560 scope.go:117] "RemoveContainer" containerID="1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150" Sep 10 15:01:56.653926 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.653885 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:56.654033 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.653985 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:56.654101 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.654032 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:01:56.654961 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.654937 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" event={"ID":"abb72069-209b-4ce9-bdd9-55f44f034466","Type":"ContainerStarted","Data":"f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb"} Sep 10 15:01:56.656058 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.656037 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:56.662272 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.662247 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:01:56.690158 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.690090 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:01:56.690366 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.690345 2560 patch_prober.go:28] interesting pod/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:01:56.690439 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:56.690399 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:01:57.677323 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:57.677280 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" event={"ID":"0cd23311-61e5-4d82-b151-11f175272b52","Type":"ContainerStarted","Data":"c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e"} Sep 10 15:01:57.689998 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:57.689963 2560 patch_prober.go:28] interesting pod/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:01:57.690175 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:57.690022 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:01:58.368780 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:58.368734 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:01:58.369103 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:58.368860 2560 patch_prober.go:28] interesting pod/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 10 15:01:58.369296 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:58.369263 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 10 15:01:58.689763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:58.689664 2560 patch_prober.go:28] interesting pod/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:01:58.689763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:58.689730 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:01:59.368630 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:59.368587 2560 patch_prober.go:28] interesting pod/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 10 15:01:59.368844 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:59.368654 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 10 15:01:59.693635 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:59.693546 2560 patch_prober.go:28] interesting pod/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:01:59.693635 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:01:59.693607 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:02:00.369222 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:00.369183 2560 patch_prober.go:28] interesting pod/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 10 15:02:00.369426 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:00.369252 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 10 15:02:00.690164 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:00.690065 2560 patch_prober.go:28] interesting pod/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:02:00.690164 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:00.690135 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:02:01.369507 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:01.369473 2560 patch_prober.go:28] interesting pod/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 10 15:02:01.370107 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:01.370074 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 10 15:02:01.725294 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:01.725215 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:02:01.725651 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:01.725617 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:02:01.727181 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:01.727161 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:02:02.369245 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:02.369204 2560 patch_prober.go:28] interesting pod/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 10 15:02:02.369536 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:02.369507 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 10 15:02:03.373676 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:03.373638 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:02:03.374102 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:03.374084 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:02:03.374861 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:03.374845 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:02:08.870819 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:08.870539 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k"] Sep 10 15:02:08.870819 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:08.870772 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" podUID="5c46aa96-e197-40c6-bc4a-3599571d86b3" containerName="maas-api" containerID="cri-o://4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b" gracePeriod=30 Sep 10 15:02:09.145052 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.145029 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:02:09.320871 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.320840 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls\") pod \"5c46aa96-e197-40c6-bc4a-3599571d86b3\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " Sep 10 15:02:09.321053 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.320917 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c46aa96-e197-40c6-bc4a-3599571d86b3-openshift-service-ca\") pod \"5c46aa96-e197-40c6-bc4a-3599571d86b3\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " Sep 10 15:02:09.321053 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.320965 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-metrics-certs\") pod \"5c46aa96-e197-40c6-bc4a-3599571d86b3\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " Sep 10 15:02:09.321053 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.321002 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k2hk\" (UniqueName: \"kubernetes.io/projected/5c46aa96-e197-40c6-bc4a-3599571d86b3-kube-api-access-9k2hk\") pod \"5c46aa96-e197-40c6-bc4a-3599571d86b3\" (UID: \"5c46aa96-e197-40c6-bc4a-3599571d86b3\") " Sep 10 15:02:09.321250 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.321220 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c46aa96-e197-40c6-bc4a-3599571d86b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c46aa96-e197-40c6-bc4a-3599571d86b3" (UID: "5c46aa96-e197-40c6-bc4a-3599571d86b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:09.321433 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.321411 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c46aa96-e197-40c6-bc4a-3599571d86b3-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:09.323086 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.323060 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c46aa96-e197-40c6-bc4a-3599571d86b3" (UID: "5c46aa96-e197-40c6-bc4a-3599571d86b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:02:09.323210 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.323194 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c46aa96-e197-40c6-bc4a-3599571d86b3-kube-api-access-9k2hk" (OuterVolumeSpecName: "kube-api-access-9k2hk") pod "5c46aa96-e197-40c6-bc4a-3599571d86b3" (UID: "5c46aa96-e197-40c6-bc4a-3599571d86b3"). InnerVolumeSpecName "kube-api-access-9k2hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:09.323435 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.323400 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c46aa96-e197-40c6-bc4a-3599571d86b3" (UID: "5c46aa96-e197-40c6-bc4a-3599571d86b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:02:09.373586 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.373501 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd"] Sep 10 15:02:09.374063 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.374010 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" containerID="cri-o://c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e" gracePeriod=30 Sep 10 15:02:09.421924 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.421894 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:09.421924 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.421918 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c46aa96-e197-40c6-bc4a-3599571d86b3-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:09.421924 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.421927 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k2hk\" (UniqueName: \"kubernetes.io/projected/5c46aa96-e197-40c6-bc4a-3599571d86b3-kube-api-access-9k2hk\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:09.751127 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.751091 2560 generic.go:356] "Generic (PLEG): container finished" podID="5c46aa96-e197-40c6-bc4a-3599571d86b3" containerID="4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b" exitCode=0 Sep 10 15:02:09.751312 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.751158 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" Sep 10 15:02:09.751312 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.751179 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" event={"ID":"5c46aa96-e197-40c6-bc4a-3599571d86b3","Type":"ContainerDied","Data":"4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b"} Sep 10 15:02:09.751312 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.751253 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k" event={"ID":"5c46aa96-e197-40c6-bc4a-3599571d86b3","Type":"ContainerDied","Data":"b5d52decc29ee54c2b0d11cc42b3a96ec0b6c43cd13cb61ca3a3c4e438b66669"} Sep 10 15:02:09.751312 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.751283 2560 scope.go:117] "RemoveContainer" containerID="4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b" Sep 10 15:02:09.784826 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.784782 2560 scope.go:117] "RemoveContainer" containerID="4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b" Sep 10 15:02:09.785110 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:02:09.785090 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b\": container with ID starting with 4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b not found: ID does not exist" containerID="4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b" Sep 10 15:02:09.785163 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.785120 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b"} err="failed to get container status \"4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b\": rpc error: code = NotFound desc = could not find container \"4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b\": container with ID starting with 4840017754f284cf55bb856a9e0eef6a201ebe12c3ead350705d6e0b0f87e13b not found: ID does not exist" Sep 10 15:02:09.797864 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.797837 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k"] Sep 10 15:02:09.816253 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:09.816230 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-24e113dd-868b956bdf-bbt2k"] Sep 10 15:02:10.399995 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.399947 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq"] Sep 10 15:02:10.401397 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.401376 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c46aa96-e197-40c6-bc4a-3599571d86b3" containerName="maas-api" Sep 10 15:02:10.401549 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.401538 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c46aa96-e197-40c6-bc4a-3599571d86b3" containerName="maas-api" Sep 10 15:02:10.401761 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.401749 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c46aa96-e197-40c6-bc4a-3599571d86b3" containerName="maas-api" Sep 10 15:02:10.412073 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.412051 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.420276 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.420245 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-12631b8d-gw-openshift-default-dockercfg-dmlxk\"" Sep 10 15:02:10.423710 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.423602 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq"] Sep 10 15:02:10.531840 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.531778 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e294e59-c965-4cca-8d17-d49086604f1a-istio-podinfo\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.532009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.531854 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-data\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.532009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.531885 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-certs\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.532009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.531913 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-envoy\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.532009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.531980 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-credential-socket\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.532153 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.532050 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-socket\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.532153 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.532086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-istio-token\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.532218 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.532187 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e294e59-c965-4cca-8d17-d49086604f1a-istiod-ca-cert\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.532249 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.532217 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfpkz\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-kube-api-access-rfpkz\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633036 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633004 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e294e59-c965-4cca-8d17-d49086604f1a-istiod-ca-cert\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633036 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfpkz\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-kube-api-access-rfpkz\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633288 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633077 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e294e59-c965-4cca-8d17-d49086604f1a-istio-podinfo\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633288 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633093 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-data\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633288 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633112 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-certs\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633288 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633137 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-envoy\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633288 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633190 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-credential-socket\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633288 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-socket\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633288 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633280 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-istio-token\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633674 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633556 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-data\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.633917 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e294e59-c965-4cca-8d17-d49086604f1a-istiod-ca-cert\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.634030 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-credential-socket\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.634030 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.633934 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-certs\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.634104 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.634041 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-socket\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.636024 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.635997 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-envoy\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.636232 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.636210 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e294e59-c965-4cca-8d17-d49086604f1a-istio-podinfo\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.645745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.645717 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-istio-token\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.646131 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.646112 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfpkz\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-kube-api-access-rfpkz\") pod \"e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:10.727177 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:10.727094 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:11.037583 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:11.037559 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq"] Sep 10 15:02:11.039848 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:02:11.039823 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e294e59_c965_4cca_8d17_d49086604f1a.slice/crio-51ea84c47082f6f9a7c56b4fb4eb1c8a73fb1624167002b0600736c53a653f0d WatchSource:0}: Error finding container 51ea84c47082f6f9a7c56b4fb4eb1c8a73fb1624167002b0600736c53a653f0d: Status 404 returned error can't find the container with id 51ea84c47082f6f9a7c56b4fb4eb1c8a73fb1624167002b0600736c53a653f0d Sep 10 15:02:11.041880 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:11.041850 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:11.041949 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:11.041912 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:11.041949 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:11.041937 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:11.669620 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:11.669564 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c46aa96-e197-40c6-bc4a-3599571d86b3" path="/var/lib/kubelet/pods/5c46aa96-e197-40c6-bc4a-3599571d86b3/volumes" Sep 10 15:02:11.790355 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:11.790313 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" event={"ID":"1e294e59-c965-4cca-8d17-d49086604f1a","Type":"ContainerStarted","Data":"e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d"} Sep 10 15:02:11.790550 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:11.790369 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" event={"ID":"1e294e59-c965-4cca-8d17-d49086604f1a","Type":"ContainerStarted","Data":"51ea84c47082f6f9a7c56b4fb4eb1c8a73fb1624167002b0600736c53a653f0d"} Sep 10 15:02:11.831848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:11.831787 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" podStartSLOduration=1.831768196 podStartE2EDuration="1.831768196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:11.829183372 +0000 UTC m=+870.643766165" watchObservedRunningTime="2026-09-10 15:02:11.831768196 +0000 UTC m=+870.646350947" Sep 10 15:02:12.728193 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:12.728158 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:12.733092 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:12.733069 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:12.795115 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:12.795087 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:12.796027 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:12.796006 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:14.646413 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.646351 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:02:14.787622 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.787581 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-istio-token\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.787622 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.787618 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-envoy\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.787919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.787637 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd23311-61e5-4d82-b151-11f175272b52-istiod-ca-cert\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.787919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.787851 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd23311-61e5-4d82-b151-11f175272b52-istio-podinfo\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.787919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.787890 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-credential-socket\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.788100 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.787944 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-data\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.788100 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.787990 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrzqv\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-kube-api-access-jrzqv\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.788100 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.788041 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd23311-61e5-4d82-b151-11f175272b52-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:14.788366 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.788213 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.788235 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.788377 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.788055 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-socket\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.788588 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-certs\") pod \"0cd23311-61e5-4d82-b151-11f175272b52\" (UID: \"0cd23311-61e5-4d82-b151-11f175272b52\") " Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.788794 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.789040 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.789062 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.789078 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.789091 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:14.789124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.789103 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd23311-61e5-4d82-b151-11f175272b52-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:14.790199 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.790177 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:14.790275 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.790217 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cd23311-61e5-4d82-b151-11f175272b52-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:02:14.790275 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.790216 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:14.790400 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.790372 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-kube-api-access-jrzqv" (OuterVolumeSpecName: "kube-api-access-jrzqv") pod "0cd23311-61e5-4d82-b151-11f175272b52" (UID: "0cd23311-61e5-4d82-b151-11f175272b52"). InnerVolumeSpecName "kube-api-access-jrzqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:14.815362 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.815331 2560 generic.go:356] "Generic (PLEG): container finished" podID="0cd23311-61e5-4d82-b151-11f175272b52" containerID="c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e" exitCode=0 Sep 10 15:02:14.815514 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.815409 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" Sep 10 15:02:14.815514 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.815417 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" event={"ID":"0cd23311-61e5-4d82-b151-11f175272b52","Type":"ContainerDied","Data":"c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e"} Sep 10 15:02:14.815514 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.815504 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd" event={"ID":"0cd23311-61e5-4d82-b151-11f175272b52","Type":"ContainerDied","Data":"70acd1ca69eb680283c44412a282361efe2e2aaa5d497f9a6deecc4b3c43beae"} Sep 10 15:02:14.815670 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.815528 2560 scope.go:117] "RemoveContainer" containerID="c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e" Sep 10 15:02:14.826500 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.826480 2560 scope.go:117] "RemoveContainer" containerID="1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150" Sep 10 15:02:14.837341 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.837248 2560 scope.go:117] "RemoveContainer" containerID="c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e" Sep 10 15:02:14.837658 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:02:14.837632 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e\": container with ID starting with c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e not found: ID does not exist" containerID="c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e" Sep 10 15:02:14.837754 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.837669 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e"} err="failed to get container status \"c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e\": rpc error: code = NotFound desc = could not find container \"c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e\": container with ID starting with c09364912dde399736d420a40531d77c275a2dbdd1eaa13acb7b2596d05df08e not found: ID does not exist" Sep 10 15:02:14.837754 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.837693 2560 scope.go:117] "RemoveContainer" containerID="1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150" Sep 10 15:02:14.838017 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:02:14.837999 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150\": container with ID starting with 1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150 not found: ID does not exist" containerID="1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150" Sep 10 15:02:14.838082 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.838024 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150"} err="failed to get container status \"1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150\": rpc error: code = NotFound desc = could not find container \"1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150\": container with ID starting with 1b2be56999e83379232fe8fb0cad1b06fa02b290430f7f545a442c8145ead150 not found: ID does not exist" Sep 10 15:02:14.840074 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.840047 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd"] Sep 10 15:02:14.845960 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.845938 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-24e113dd-openshift-default-9bc4cc46d-tnkgd"] Sep 10 15:02:14.889820 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.889766 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd23311-61e5-4d82-b151-11f175272b52-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:14.889919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.889826 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrzqv\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-kube-api-access-jrzqv\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:14.889919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.889843 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd23311-61e5-4d82-b151-11f175272b52-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:14.889919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:14.889857 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd23311-61e5-4d82-b151-11f175272b52-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:15.666375 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:15.666346 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd23311-61e5-4d82-b151-11f175272b52" path="/var/lib/kubelet/pods/0cd23311-61e5-4d82-b151-11f175272b52/volumes" Sep 10 15:02:16.829715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:16.829637 2560 generic.go:356] "Generic (PLEG): container finished" podID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerID="78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec" exitCode=0 Sep 10 15:02:16.829715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:16.829685 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" event={"ID":"0dc0e1a4-20b3-4144-b4e1-382919c300da","Type":"ContainerDied","Data":"78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec"} Sep 10 15:02:16.830161 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:16.830124 2560 scope.go:117] "RemoveContainer" containerID="78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec" Sep 10 15:02:16.831184 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:16.831158 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:16.831262 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:16.831237 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:16.831298 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:16.831274 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:17.837109 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:17.837073 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" event={"ID":"0dc0e1a4-20b3-4144-b4e1-382919c300da","Type":"ContainerStarted","Data":"634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c"} Sep 10 15:02:18.439130 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:18.439097 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:02:18.439337 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:18.439302 2560 patch_prober.go:28] interesting pod/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:02:18.439422 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:18.439370 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:02:19.439319 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:19.439277 2560 patch_prober.go:28] interesting pod/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:02:19.439705 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:19.439346 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:02:20.438743 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:20.438703 2560 patch_prober.go:28] interesting pod/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:02:20.438917 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:20.438762 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:02:21.438753 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:21.438718 2560 patch_prober.go:28] interesting pod/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:02:21.439171 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:21.438789 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:02:22.439201 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:22.439162 2560 patch_prober.go:28] interesting pod/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:02:22.439586 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:22.439237 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:02:23.447299 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:23.447263 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:02:23.447775 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:23.447633 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:02:23.449236 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:23.449210 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:02:33.927021 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:33.926988 2560 generic.go:356] "Generic (PLEG): container finished" podID="1e294e59-c965-4cca-8d17-d49086604f1a" containerID="e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d" exitCode=0 Sep 10 15:02:33.927021 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:33.927029 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" event={"ID":"1e294e59-c965-4cca-8d17-d49086604f1a","Type":"ContainerDied","Data":"e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d"} Sep 10 15:02:33.927484 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:33.927399 2560 scope.go:117] "RemoveContainer" containerID="e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d" Sep 10 15:02:33.928442 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:33.928412 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:33.928491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:33.928481 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:33.928527 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:33.928513 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:34.933457 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:34.933423 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" event={"ID":"1e294e59-c965-4cca-8d17-d49086604f1a","Type":"ContainerStarted","Data":"e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296"} Sep 10 15:02:35.727930 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:35.727897 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:35.728112 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:35.728089 2560 patch_prober.go:28] interesting pod/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 15:02:35.728165 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:35.728137 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 15:02:36.727848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:36.727787 2560 patch_prober.go:28] interesting pod/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 15:02:36.728233 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:36.727878 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 15:02:37.727874 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:37.727840 2560 patch_prober.go:28] interesting pod/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 15:02:37.728264 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:37.727918 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 15:02:38.727680 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:38.727642 2560 patch_prober.go:28] interesting pod/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 15:02:38.727899 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:38.727697 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 15:02:39.728403 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:39.728371 2560 patch_prober.go:28] interesting pod/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 10 15:02:39.728767 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:39.728422 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 10 15:02:40.732212 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:40.732182 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:40.732691 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:40.732668 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:40.733325 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:40.733306 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:41.646580 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:41.646548 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:02:41.647057 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:41.647038 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:02:45.556829 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:45.554737 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq"] Sep 10 15:02:45.556829 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:45.555103 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" containerID="cri-o://e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296" gracePeriod=30 Sep 10 15:02:46.310917 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.310870 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5"] Sep 10 15:02:46.311377 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.311360 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" Sep 10 15:02:46.311377 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.311378 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" Sep 10 15:02:46.311520 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.311390 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" Sep 10 15:02:46.311520 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.311399 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" Sep 10 15:02:46.311520 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.311488 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" Sep 10 15:02:46.311520 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.311496 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd23311-61e5-4d82-b151-11f175272b52" containerName="istio-proxy" Sep 10 15:02:46.315925 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.315906 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.318497 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.318476 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-78cfd1dc-openshift-default-dockercfg-lmxbn\"" Sep 10 15:02:46.329356 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.329332 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5"] Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.403593 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-data\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.403664 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-credential-socket\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.403723 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istiod-ca-cert\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.403777 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-token\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.403940 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-socket\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.403992 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-envoy\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.404019 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-podinfo\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.404059 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk6mf\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-kube-api-access-gk6mf\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.406491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.404097 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-certs\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505364 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505331 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-token\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505389 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-socket\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-envoy\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505444 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-podinfo\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505469 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk6mf\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-kube-api-access-gk6mf\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-certs\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505532 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-data\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505562 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-credential-socket\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505988 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505621 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istiod-ca-cert\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.505988 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.505860 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-socket\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.506175 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.506154 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-certs\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.506244 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.506212 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-data\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.506353 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.506335 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-credential-socket\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.506353 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.506344 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istiod-ca-cert\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.507915 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.507890 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-envoy\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.508139 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.508121 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-podinfo\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.514381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.514352 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-token\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.514381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.514376 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk6mf\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-kube-api-access-gk6mf\") pod \"e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.625993 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.625969 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:46.904673 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.904647 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5"] Sep 10 15:02:46.906771 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:02:46.906743 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd0fd4b_9de1_463f_94ac_efbf1919a28e.slice/crio-b8eb5810a0e1335a07021892f603fb828f67a21175fed3f4198cfad0a163c800 WatchSource:0}: Error finding container b8eb5810a0e1335a07021892f603fb828f67a21175fed3f4198cfad0a163c800: Status 404 returned error can't find the container with id b8eb5810a0e1335a07021892f603fb828f67a21175fed3f4198cfad0a163c800 Sep 10 15:02:46.909390 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.909352 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:46.909490 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.909448 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:46.909490 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.909480 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:02:46.995901 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.995861 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" event={"ID":"5bd0fd4b-9de1-463f-94ac-efbf1919a28e","Type":"ContainerStarted","Data":"b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af"} Sep 10 15:02:46.996046 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:46.995939 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" event={"ID":"5bd0fd4b-9de1-463f-94ac-efbf1919a28e","Type":"ContainerStarted","Data":"b8eb5810a0e1335a07021892f603fb828f67a21175fed3f4198cfad0a163c800"} Sep 10 15:02:47.018032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:47.017984 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" podStartSLOduration=1.017968501 podStartE2EDuration="1.017968501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:47.0151741 +0000 UTC m=+905.829756896" watchObservedRunningTime="2026-09-10 15:02:47.017968501 +0000 UTC m=+905.832551252" Sep 10 15:02:47.626981 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:47.626948 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:47.632178 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:47.632152 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:48.000757 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:48.000678 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:48.001712 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:48.001695 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:02:49.477117 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.477083 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4"] Sep 10 15:02:49.481367 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.481345 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.483755 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.483734 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-78cfd1dc-serving-cert\"" Sep 10 15:02:49.493824 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.493781 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4"] Sep 10 15:02:49.637911 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.637883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-metrics-certs\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.638088 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.637936 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-openshift-service-ca\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.638088 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.637982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-maas-api-tls\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.638088 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.637998 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdcj\" (UniqueName: \"kubernetes.io/projected/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-kube-api-access-6fdcj\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.739311 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.739232 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-metrics-certs\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.739311 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.739294 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-openshift-service-ca\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.739531 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.739322 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-maas-api-tls\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.739531 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.739340 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdcj\" (UniqueName: \"kubernetes.io/projected/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-kube-api-access-6fdcj\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.740044 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.740019 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-openshift-service-ca\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.741744 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.741723 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-metrics-certs\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.741868 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.741777 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-maas-api-tls\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.746986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.746957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdcj\" (UniqueName: \"kubernetes.io/projected/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-kube-api-access-6fdcj\") pod \"maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:49.794868 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:49.794843 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:50.097279 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.097253 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4"] Sep 10 15:02:50.098506 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:02:50.098477 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6f5adf_32fe_4bb9_9345_82d609fcf2d7.slice/crio-ad94981f655af5b04ffcd4d6adf1f56b7ba28a860391c82c2ab8159ca67950d8 WatchSource:0}: Error finding container ad94981f655af5b04ffcd4d6adf1f56b7ba28a860391c82c2ab8159ca67950d8: Status 404 returned error can't find the container with id ad94981f655af5b04ffcd4d6adf1f56b7ba28a860391c82c2ab8159ca67950d8 Sep 10 15:02:50.810787 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.810765 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:50.956514 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956484 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-data\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.956514 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956518 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-socket\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.956784 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956552 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e294e59-c965-4cca-8d17-d49086604f1a-istio-podinfo\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.956784 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956572 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-istio-token\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.956784 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956599 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e294e59-c965-4cca-8d17-d49086604f1a-istiod-ca-cert\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.956784 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956631 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-certs\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.956784 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956705 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-envoy\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.956784 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956742 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfpkz\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-kube-api-access-rfpkz\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.956784 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956780 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-credential-socket\") pod \"1e294e59-c965-4cca-8d17-d49086604f1a\" (UID: \"1e294e59-c965-4cca-8d17-d49086604f1a\") " Sep 10 15:02:50.957132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956826 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:50.957132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.956855 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:50.957132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.957030 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:50.957132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.957074 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:50.957348 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.957296 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:50.957348 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.957316 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:50.957348 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.957328 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:50.957348 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.957340 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:50.957511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.957345 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e294e59-c965-4cca-8d17-d49086604f1a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:02:50.958861 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.958829 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e294e59-c965-4cca-8d17-d49086604f1a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:02:50.958977 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.958864 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:02:50.959174 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.959151 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-kube-api-access-rfpkz" (OuterVolumeSpecName: "kube-api-access-rfpkz") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "kube-api-access-rfpkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:50.959257 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:50.959192 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e294e59-c965-4cca-8d17-d49086604f1a" (UID: "1e294e59-c965-4cca-8d17-d49086604f1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:02:51.016669 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.016640 2560 generic.go:356] "Generic (PLEG): container finished" podID="1e294e59-c965-4cca-8d17-d49086604f1a" containerID="e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296" exitCode=0 Sep 10 15:02:51.016827 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.016687 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" Sep 10 15:02:51.016827 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.016716 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" event={"ID":"1e294e59-c965-4cca-8d17-d49086604f1a","Type":"ContainerDied","Data":"e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296"} Sep 10 15:02:51.016827 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.016772 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq" event={"ID":"1e294e59-c965-4cca-8d17-d49086604f1a","Type":"ContainerDied","Data":"51ea84c47082f6f9a7c56b4fb4eb1c8a73fb1624167002b0600736c53a653f0d"} Sep 10 15:02:51.016827 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.016806 2560 scope.go:117] "RemoveContainer" containerID="e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296" Sep 10 15:02:51.018607 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.018573 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" event={"ID":"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7","Type":"ContainerStarted","Data":"4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428"} Sep 10 15:02:51.018720 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.018640 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:02:51.018720 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.018665 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" event={"ID":"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7","Type":"ContainerStarted","Data":"ad94981f655af5b04ffcd4d6adf1f56b7ba28a860391c82c2ab8159ca67950d8"} Sep 10 15:02:51.026786 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.026770 2560 scope.go:117] "RemoveContainer" containerID="e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d" Sep 10 15:02:51.034392 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.034371 2560 scope.go:117] "RemoveContainer" containerID="e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296" Sep 10 15:02:51.034664 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:02:51.034647 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296\": container with ID starting with e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296 not found: ID does not exist" containerID="e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296" Sep 10 15:02:51.034726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.034671 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296"} err="failed to get container status \"e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296\": rpc error: code = NotFound desc = could not find container \"e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296\": container with ID starting with e84a6e24ad5ce303122bcef360aa5c1054d2b430f5e379b23c62595d2ad05296 not found: ID does not exist" Sep 10 15:02:51.034726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.034688 2560 scope.go:117] "RemoveContainer" containerID="e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d" Sep 10 15:02:51.034970 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:02:51.034953 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d\": container with ID starting with e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d not found: ID does not exist" containerID="e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d" Sep 10 15:02:51.035021 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.034976 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d"} err="failed to get container status \"e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d\": rpc error: code = NotFound desc = could not find container \"e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d\": container with ID starting with e327e4ea0d9326bcef81fdd1519c5769a7716d3009e7adfbbea7a6f881fc6f2d not found: ID does not exist" Sep 10 15:02:51.036583 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.036547 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" podStartSLOduration=1.514307193 podStartE2EDuration="2.036535092s" totalImagesPullingTime="522.227899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:51.035625035 +0000 UTC m=+909.850207786" watchObservedRunningTime="2026-09-10 15:02:51.036535092 +0000 UTC m=+909.851117843" Sep 10 15:02:51.055495 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.055475 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq"] Sep 10 15:02:51.058040 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.058021 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e294e59-c965-4cca-8d17-d49086604f1a-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:51.058040 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.058041 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:51.058159 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.058049 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e294e59-c965-4cca-8d17-d49086604f1a-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:51.058159 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.058057 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e294e59-c965-4cca-8d17-d49086604f1a-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:51.058159 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.058065 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfpkz\" (UniqueName: \"kubernetes.io/projected/1e294e59-c965-4cca-8d17-d49086604f1a-kube-api-access-rfpkz\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:02:51.061246 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.061225 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-12631b8d-gw-openshift-default-747c4f668-d7bhq"] Sep 10 15:02:51.664532 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:51.664498 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" path="/var/lib/kubelet/pods/1e294e59-c965-4cca-8d17-d49086604f1a/volumes" Sep 10 15:02:57.031447 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:02:57.031421 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:03:13.132531 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:13.132500 2560 generic.go:356] "Generic (PLEG): container finished" podID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerID="b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af" exitCode=0 Sep 10 15:03:13.132880 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:13.132556 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" event={"ID":"5bd0fd4b-9de1-463f-94ac-efbf1919a28e","Type":"ContainerDied","Data":"b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af"} Sep 10 15:03:13.132974 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:13.132958 2560 scope.go:117] "RemoveContainer" containerID="b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af" Sep 10 15:03:13.134043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:13.134015 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:13.134116 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:13.134083 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:13.134116 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:13.134112 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:14.139532 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:14.139497 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" event={"ID":"5bd0fd4b-9de1-463f-94ac-efbf1919a28e","Type":"ContainerStarted","Data":"1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c"} Sep 10 15:03:14.627300 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:14.627255 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:03:14.628261 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:14.628234 2560 patch_prober.go:28] interesting pod/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:03:14.628380 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:14.628285 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:03:15.627087 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:15.627048 2560 patch_prober.go:28] interesting pod/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:03:15.627468 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:15.627108 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:03:16.627231 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:16.627186 2560 patch_prober.go:28] interesting pod/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:03:16.627716 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:16.627257 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:03:17.626372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:17.626339 2560 patch_prober.go:28] interesting pod/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:03:17.626549 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:17.626390 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:03:18.001637 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:18.001542 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:03:18.627161 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:18.627127 2560 patch_prober.go:28] interesting pod/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:03:18.627331 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:18.627182 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:03:19.630313 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:19.630282 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:03:19.631343 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:19.631323 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:03:20.368587 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.368551 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4"] Sep 10 15:03:20.368876 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.368847 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" podUID="5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" containerName="maas-api" containerID="cri-o://4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428" gracePeriod=30 Sep 10 15:03:20.617724 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.617701 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:03:20.763820 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.763407 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-openshift-service-ca\") pod \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " Sep 10 15:03:20.763820 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.763513 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-maas-api-tls\") pod \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " Sep 10 15:03:20.763820 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.763583 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fdcj\" (UniqueName: \"kubernetes.io/projected/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-kube-api-access-6fdcj\") pod \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " Sep 10 15:03:20.763820 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.763618 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-metrics-certs\") pod \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\" (UID: \"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7\") " Sep 10 15:03:20.769235 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.769187 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" (UID: "5a6f5adf-32fe-4bb9-9345-82d609fcf2d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:20.773394 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.773348 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" (UID: "5a6f5adf-32fe-4bb9-9345-82d609fcf2d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:03:20.773657 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.773614 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" (UID: "5a6f5adf-32fe-4bb9-9345-82d609fcf2d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:03:20.782889 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.782019 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-kube-api-access-6fdcj" (OuterVolumeSpecName: "kube-api-access-6fdcj") pod "5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" (UID: "5a6f5adf-32fe-4bb9-9345-82d609fcf2d7"). InnerVolumeSpecName "kube-api-access-6fdcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:20.865404 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.865374 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:20.865620 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.865412 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:20.865620 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.865422 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fdcj\" (UniqueName: \"kubernetes.io/projected/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-kube-api-access-6fdcj\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:20.865620 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:20.865431 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:21.191695 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.191606 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" containerID="4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428" exitCode=0 Sep 10 15:03:21.191695 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.191648 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" Sep 10 15:03:21.191695 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.191670 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" event={"ID":"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7","Type":"ContainerDied","Data":"4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428"} Sep 10 15:03:21.191950 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.191720 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4" event={"ID":"5a6f5adf-32fe-4bb9-9345-82d609fcf2d7","Type":"ContainerDied","Data":"ad94981f655af5b04ffcd4d6adf1f56b7ba28a860391c82c2ab8159ca67950d8"} Sep 10 15:03:21.191950 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.191742 2560 scope.go:117] "RemoveContainer" containerID="4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428" Sep 10 15:03:21.200705 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.200684 2560 scope.go:117] "RemoveContainer" containerID="4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428" Sep 10 15:03:21.201045 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:03:21.201027 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428\": container with ID starting with 4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428 not found: ID does not exist" containerID="4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428" Sep 10 15:03:21.201097 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.201055 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428"} err="failed to get container status \"4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428\": rpc error: code = NotFound desc = could not find container \"4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428\": container with ID starting with 4c52375b1561ac246ac3627322c89002b9284bf245af22aff6c781299f777428 not found: ID does not exist" Sep 10 15:03:21.212967 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.212942 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4"] Sep 10 15:03:21.217167 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.217145 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-78cfd1dc-588b474d49-6rtg4"] Sep 10 15:03:21.667766 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.667729 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" path="/var/lib/kubelet/pods/5a6f5adf-32fe-4bb9-9345-82d609fcf2d7/volumes" Sep 10 15:03:21.687017 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.686983 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5"] Sep 10 15:03:21.687211 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:21.687177 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" containerID="cri-o://1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c" gracePeriod=30 Sep 10 15:03:22.406080 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406041 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln"] Sep 10 15:03:22.406636 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406618 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" Sep 10 15:03:22.406725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406640 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" Sep 10 15:03:22.406725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406659 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" Sep 10 15:03:22.406725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406669 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" Sep 10 15:03:22.406725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406693 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" containerName="maas-api" Sep 10 15:03:22.406725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406701 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" containerName="maas-api" Sep 10 15:03:22.407019 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406819 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" Sep 10 15:03:22.407019 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406834 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e294e59-c965-4cca-8d17-d49086604f1a" containerName="istio-proxy" Sep 10 15:03:22.407019 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.406848 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6f5adf-32fe-4bb9-9345-82d609fcf2d7" containerName="maas-api" Sep 10 15:03:22.411421 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.411400 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.414252 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.414234 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c3f76b1a-openshift-default-dockercfg-2tjng\"" Sep 10 15:03:22.426034 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.426010 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln"] Sep 10 15:03:22.583433 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583390 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-data\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.583621 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583474 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-envoy\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.583621 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583514 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-podinfo\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.583621 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583563 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-socket\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.583793 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583726 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-credential-socket\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.583793 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583762 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfs8s\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-kube-api-access-mfs8s\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.583925 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583886 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-token\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.583925 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583914 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istiod-ca-cert\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.584001 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.583940 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-certs\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685144 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685048 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-envoy\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685144 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685102 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-podinfo\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685388 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685150 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-socket\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685388 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685234 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-credential-socket\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685388 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfs8s\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-kube-api-access-mfs8s\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685388 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685374 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-token\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685594 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685492 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istiod-ca-cert\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685594 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685557 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-certs\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685695 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685583 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-credential-socket\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685695 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685608 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-socket\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.685695 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685594 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-data\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.686112 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685919 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-data\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.686234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.685926 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-certs\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.693828 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.687002 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istiod-ca-cert\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.694504 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.694480 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-podinfo\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.694642 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.694536 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-envoy\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.696142 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.696116 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfs8s\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-kube-api-access-mfs8s\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.696630 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.696608 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-token\") pod \"e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.722027 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.722006 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:22.997215 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:22.997190 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln"] Sep 10 15:03:22.998865 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:03:22.998835 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dc16fc_089f_447a_bfa3_3bc6c4bd911c.slice/crio-ab938fb14c1aa3b552156f466e9cd7c20d5682f7732d754c114c1498f1afc5dd WatchSource:0}: Error finding container ab938fb14c1aa3b552156f466e9cd7c20d5682f7732d754c114c1498f1afc5dd: Status 404 returned error can't find the container with id ab938fb14c1aa3b552156f466e9cd7c20d5682f7732d754c114c1498f1afc5dd Sep 10 15:03:23.003975 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:23.003941 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:23.004055 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:23.004008 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:23.004055 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:23.004036 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:23.205039 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:23.205001 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" event={"ID":"10dc16fc-089f-447a-bfa3-3bc6c4bd911c","Type":"ContainerStarted","Data":"70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee"} Sep 10 15:03:23.205203 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:23.205064 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" event={"ID":"10dc16fc-089f-447a-bfa3-3bc6c4bd911c","Type":"ContainerStarted","Data":"ab938fb14c1aa3b552156f466e9cd7c20d5682f7732d754c114c1498f1afc5dd"} Sep 10 15:03:23.229192 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:23.229147 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" podStartSLOduration=1.229132853 podStartE2EDuration="1.229132853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:23.226775266 +0000 UTC m=+942.041358017" watchObservedRunningTime="2026-09-10 15:03:23.229132853 +0000 UTC m=+942.043715664" Sep 10 15:03:23.723023 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:23.722989 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:24.727517 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:24.727488 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:25.214613 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.214585 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:25.215529 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.215506 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:25.691819 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.691398 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2"] Sep 10 15:03:25.703816 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.702207 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.713829 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.711339 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c3f76b1a-serving-cert\"" Sep 10 15:03:25.724943 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.724913 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2"] Sep 10 15:03:25.826858 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.825994 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542fe8bc-53f3-4020-a112-e5753f00aced-openshift-service-ca\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.826858 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.826126 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-maas-api-tls\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.826858 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.826233 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-metrics-certs\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.827445 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.826961 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfx8q\" (UniqueName: \"kubernetes.io/projected/542fe8bc-53f3-4020-a112-e5753f00aced-kube-api-access-tfx8q\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.927567 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.927537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-metrics-certs\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.927744 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.927619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfx8q\" (UniqueName: \"kubernetes.io/projected/542fe8bc-53f3-4020-a112-e5753f00aced-kube-api-access-tfx8q\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.927744 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.927644 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542fe8bc-53f3-4020-a112-e5753f00aced-openshift-service-ca\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.927744 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.927682 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-maas-api-tls\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.928327 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.928303 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542fe8bc-53f3-4020-a112-e5753f00aced-openshift-service-ca\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.929998 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.929978 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-maas-api-tls\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.930085 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.930004 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-metrics-certs\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:25.935657 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:25.935637 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfx8q\" (UniqueName: \"kubernetes.io/projected/542fe8bc-53f3-4020-a112-e5753f00aced-kube-api-access-tfx8q\") pod \"maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:26.039763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:26.039672 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:26.331704 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:26.331673 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2"] Sep 10 15:03:26.332434 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:03:26.332373 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542fe8bc_53f3_4020_a112_e5753f00aced.slice/crio-b8a112c4ba189669b18e17ca435a5ab6a7bbda1a2df6fbcb78c92274f5ca6ec8 WatchSource:0}: Error finding container b8a112c4ba189669b18e17ca435a5ab6a7bbda1a2df6fbcb78c92274f5ca6ec8: Status 404 returned error can't find the container with id b8a112c4ba189669b18e17ca435a5ab6a7bbda1a2df6fbcb78c92274f5ca6ec8 Sep 10 15:03:26.333383 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:26.333364 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:03:26.921218 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:26.921192 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:03:27.038756 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038687 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-envoy\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.038756 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038723 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-podinfo\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.038990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038759 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-data\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.038990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038778 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-credential-socket\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.038990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038809 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-certs\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.038990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038834 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istiod-ca-cert\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.038990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038863 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-socket\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.038990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038920 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-token\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.038990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.038970 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk6mf\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-kube-api-access-gk6mf\") pod \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\" (UID: \"5bd0fd4b-9de1-463f-94ac-efbf1919a28e\") " Sep 10 15:03:27.039352 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.039063 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-data" (OuterVolumeSpecName: "istio-data") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:03:27.039352 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.039074 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:03:27.039352 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.039201 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:03:27.039352 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.039321 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:03:27.039561 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.039390 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.039561 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.039410 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.039561 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.039411 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:27.039561 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.039425 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.040950 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.040925 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:03:27.041083 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.041052 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:03:27.041158 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.041139 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-kube-api-access-gk6mf" (OuterVolumeSpecName: "kube-api-access-gk6mf") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "kube-api-access-gk6mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:27.041198 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.041166 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bd0fd4b-9de1-463f-94ac-efbf1919a28e" (UID: "5bd0fd4b-9de1-463f-94ac-efbf1919a28e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:27.140978 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.140944 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.140978 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.140974 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk6mf\" (UniqueName: \"kubernetes.io/projected/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-kube-api-access-gk6mf\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.141166 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.140989 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.141166 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.141001 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.141166 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.141011 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.141166 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.141021 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd0fd4b-9de1-463f-94ac-efbf1919a28e-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:27.226028 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.225994 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" event={"ID":"542fe8bc-53f3-4020-a112-e5753f00aced","Type":"ContainerStarted","Data":"933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768"} Sep 10 15:03:27.226201 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.226091 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" event={"ID":"542fe8bc-53f3-4020-a112-e5753f00aced","Type":"ContainerStarted","Data":"b8a112c4ba189669b18e17ca435a5ab6a7bbda1a2df6fbcb78c92274f5ca6ec8"} Sep 10 15:03:27.226201 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.226123 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:27.227593 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.227569 2560 generic.go:356] "Generic (PLEG): container finished" podID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerID="1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c" exitCode=0 Sep 10 15:03:27.227696 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.227606 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" Sep 10 15:03:27.227696 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.227648 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" event={"ID":"5bd0fd4b-9de1-463f-94ac-efbf1919a28e","Type":"ContainerDied","Data":"1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c"} Sep 10 15:03:27.227696 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.227682 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5" event={"ID":"5bd0fd4b-9de1-463f-94ac-efbf1919a28e","Type":"ContainerDied","Data":"b8eb5810a0e1335a07021892f603fb828f67a21175fed3f4198cfad0a163c800"} Sep 10 15:03:27.227809 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.227703 2560 scope.go:117] "RemoveContainer" containerID="1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c" Sep 10 15:03:27.237492 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.237475 2560 scope.go:117] "RemoveContainer" containerID="b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af" Sep 10 15:03:27.245303 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.245282 2560 scope.go:117] "RemoveContainer" containerID="1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c" Sep 10 15:03:27.245608 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:03:27.245589 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c\": container with ID starting with 1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c not found: ID does not exist" containerID="1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c" Sep 10 15:03:27.245659 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.245615 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c"} err="failed to get container status \"1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c\": rpc error: code = NotFound desc = could not find container \"1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c\": container with ID starting with 1af760ada44c736a44ec31a95e39b7b68ea1d7bfd1286e1e2aab912775190c6c not found: ID does not exist" Sep 10 15:03:27.245659 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.245631 2560 scope.go:117] "RemoveContainer" containerID="b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af" Sep 10 15:03:27.245909 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:03:27.245889 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af\": container with ID starting with b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af not found: ID does not exist" containerID="b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af" Sep 10 15:03:27.245953 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.245915 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af"} err="failed to get container status \"b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af\": rpc error: code = NotFound desc = could not find container \"b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af\": container with ID starting with b52c746faaefdc2a8850202553e5232df75dd6b681f98a6feef1cd4adcd0f7af not found: ID does not exist" Sep 10 15:03:27.248502 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.248463 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" podStartSLOduration=1.873813961 podStartE2EDuration="2.248450602s" totalImagesPullingTime="374.636641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:27.247127717 +0000 UTC m=+946.061710493" watchObservedRunningTime="2026-09-10 15:03:27.248450602 +0000 UTC m=+946.063033352" Sep 10 15:03:27.265179 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.265155 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5"] Sep 10 15:03:27.270075 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.270054 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-78cfd1dc-openshift-default-d8fb86b96-lqwj5"] Sep 10 15:03:27.667869 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:27.667834 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" path="/var/lib/kubelet/pods/5bd0fd4b-9de1-463f-94ac-efbf1919a28e/volumes" Sep 10 15:03:33.243250 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:33.243220 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:46.342126 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:46.342090 2560 generic.go:356] "Generic (PLEG): container finished" podID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerID="70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee" exitCode=0 Sep 10 15:03:46.342523 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:46.342163 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" event={"ID":"10dc16fc-089f-447a-bfa3-3bc6c4bd911c","Type":"ContainerDied","Data":"70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee"} Sep 10 15:03:46.342567 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:46.342533 2560 scope.go:117] "RemoveContainer" containerID="70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee" Sep 10 15:03:46.343598 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:46.343570 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:46.343646 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:46.343638 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:46.343681 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:46.343669 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:03:47.348820 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:47.348775 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" event={"ID":"10dc16fc-089f-447a-bfa3-3bc6c4bd911c","Type":"ContainerStarted","Data":"ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242"} Sep 10 15:03:47.722540 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:47.722505 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:47.722709 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:47.722672 2560 patch_prober.go:28] interesting pod/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:03:47.722753 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:47.722729 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 10 15:03:48.722402 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:48.722368 2560 patch_prober.go:28] interesting pod/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:03:48.722785 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:48.722425 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 10 15:03:49.722914 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:49.722880 2560 patch_prober.go:28] interesting pod/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:03:49.723312 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:49.722947 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 10 15:03:50.722556 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:50.722522 2560 patch_prober.go:28] interesting pod/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:03:50.722738 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:50.722581 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 10 15:03:51.722377 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:51.722341 2560 patch_prober.go:28] interesting pod/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 10 15:03:51.722876 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:51.722402 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 10 15:03:52.726911 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:52.726881 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:52.727582 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:52.727274 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:52.727939 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:52.727921 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:03:56.682484 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:56.682443 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2"] Sep 10 15:03:56.682902 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:56.682768 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" podUID="542fe8bc-53f3-4020-a112-e5753f00aced" containerName="maas-api" containerID="cri-o://933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768" gracePeriod=30 Sep 10 15:03:56.935416 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:56.935362 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:57.046135 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.046106 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-maas-api-tls\") pod \"542fe8bc-53f3-4020-a112-e5753f00aced\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " Sep 10 15:03:57.046301 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.046159 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfx8q\" (UniqueName: \"kubernetes.io/projected/542fe8bc-53f3-4020-a112-e5753f00aced-kube-api-access-tfx8q\") pod \"542fe8bc-53f3-4020-a112-e5753f00aced\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " Sep 10 15:03:57.046301 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.046183 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542fe8bc-53f3-4020-a112-e5753f00aced-openshift-service-ca\") pod \"542fe8bc-53f3-4020-a112-e5753f00aced\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " Sep 10 15:03:57.046301 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.046234 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-metrics-certs\") pod \"542fe8bc-53f3-4020-a112-e5753f00aced\" (UID: \"542fe8bc-53f3-4020-a112-e5753f00aced\") " Sep 10 15:03:57.046606 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.046576 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542fe8bc-53f3-4020-a112-e5753f00aced-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "542fe8bc-53f3-4020-a112-e5753f00aced" (UID: "542fe8bc-53f3-4020-a112-e5753f00aced"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:03:57.048285 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.048260 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "542fe8bc-53f3-4020-a112-e5753f00aced" (UID: "542fe8bc-53f3-4020-a112-e5753f00aced"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:03:57.048285 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.048278 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "542fe8bc-53f3-4020-a112-e5753f00aced" (UID: "542fe8bc-53f3-4020-a112-e5753f00aced"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:03:57.048426 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.048375 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542fe8bc-53f3-4020-a112-e5753f00aced-kube-api-access-tfx8q" (OuterVolumeSpecName: "kube-api-access-tfx8q") pod "542fe8bc-53f3-4020-a112-e5753f00aced" (UID: "542fe8bc-53f3-4020-a112-e5753f00aced"). InnerVolumeSpecName "kube-api-access-tfx8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:03:57.147809 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.147760 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:57.147809 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.147812 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfx8q\" (UniqueName: \"kubernetes.io/projected/542fe8bc-53f3-4020-a112-e5753f00aced-kube-api-access-tfx8q\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:57.148049 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.147828 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/542fe8bc-53f3-4020-a112-e5753f00aced-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:57.148049 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.147842 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542fe8bc-53f3-4020-a112-e5753f00aced-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:03:57.166492 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.166464 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln"] Sep 10 15:03:57.166719 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.166666 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" containerID="cri-o://ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242" gracePeriod=30 Sep 10 15:03:57.410447 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.410412 2560 generic.go:356] "Generic (PLEG): container finished" podID="542fe8bc-53f3-4020-a112-e5753f00aced" containerID="933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768" exitCode=0 Sep 10 15:03:57.410447 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.410448 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" event={"ID":"542fe8bc-53f3-4020-a112-e5753f00aced","Type":"ContainerDied","Data":"933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768"} Sep 10 15:03:57.410700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.410462 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" Sep 10 15:03:57.410700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.410487 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2" event={"ID":"542fe8bc-53f3-4020-a112-e5753f00aced","Type":"ContainerDied","Data":"b8a112c4ba189669b18e17ca435a5ab6a7bbda1a2df6fbcb78c92274f5ca6ec8"} Sep 10 15:03:57.410700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.410508 2560 scope.go:117] "RemoveContainer" containerID="933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768" Sep 10 15:03:57.419892 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.419875 2560 scope.go:117] "RemoveContainer" containerID="933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768" Sep 10 15:03:57.420156 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:03:57.420138 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768\": container with ID starting with 933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768 not found: ID does not exist" containerID="933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768" Sep 10 15:03:57.420204 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.420164 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768"} err="failed to get container status \"933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768\": rpc error: code = NotFound desc = could not find container \"933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768\": container with ID starting with 933ee8ee71243926b7b79bb96ece9f7e021207414a58992fed3c483dbcaa5768 not found: ID does not exist" Sep 10 15:03:57.435603 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.435576 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2"] Sep 10 15:03:57.439771 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.439750 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c3f76b1a-987c8bc4f-ztbb2"] Sep 10 15:03:57.665144 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:03:57.665067 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="542fe8bc-53f3-4020-a112-e5753f00aced" path="/var/lib/kubelet/pods/542fe8bc-53f3-4020-a112-e5753f00aced/volumes" Sep 10 15:04:02.424789 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.424763 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:04:02.437864 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.437778 2560 generic.go:356] "Generic (PLEG): container finished" podID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerID="ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242" exitCode=0 Sep 10 15:04:02.437864 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.437840 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" Sep 10 15:04:02.438009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.437873 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" event={"ID":"10dc16fc-089f-447a-bfa3-3bc6c4bd911c","Type":"ContainerDied","Data":"ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242"} Sep 10 15:04:02.438009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.437938 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln" event={"ID":"10dc16fc-089f-447a-bfa3-3bc6c4bd911c","Type":"ContainerDied","Data":"ab938fb14c1aa3b552156f466e9cd7c20d5682f7732d754c114c1498f1afc5dd"} Sep 10 15:04:02.438009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.437981 2560 scope.go:117] "RemoveContainer" containerID="ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242" Sep 10 15:04:02.448317 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.448266 2560 scope.go:117] "RemoveContainer" containerID="70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee" Sep 10 15:04:02.456423 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.456408 2560 scope.go:117] "RemoveContainer" containerID="ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242" Sep 10 15:04:02.456683 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:04:02.456665 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242\": container with ID starting with ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242 not found: ID does not exist" containerID="ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242" Sep 10 15:04:02.456751 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.456695 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242"} err="failed to get container status \"ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242\": rpc error: code = NotFound desc = could not find container \"ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242\": container with ID starting with ff0a1724c81ad0c1a31668a9a8293a20a2b7c85d9bc646d72716fc5f56669242 not found: ID does not exist" Sep 10 15:04:02.456751 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.456722 2560 scope.go:117] "RemoveContainer" containerID="70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee" Sep 10 15:04:02.457044 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:04:02.457027 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee\": container with ID starting with 70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee not found: ID does not exist" containerID="70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee" Sep 10 15:04:02.457108 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.457051 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee"} err="failed to get container status \"70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee\": rpc error: code = NotFound desc = could not find container \"70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee\": container with ID starting with 70c53efb767105c9a4d3aab5d40d8ffbf762d620ceba021217d3d61f64d93cee not found: ID does not exist" Sep 10 15:04:02.499943 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.499914 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-socket\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500118 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.499982 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istiod-ca-cert\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500118 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500008 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-credential-socket\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500118 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500034 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-data\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500118 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500060 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfs8s\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-kube-api-access-mfs8s\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500346 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500193 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-certs\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500346 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500201 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:02.500346 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500270 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-envoy\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500346 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500306 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-token\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500538 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500357 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-podinfo\") pod \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\" (UID: \"10dc16fc-089f-447a-bfa3-3bc6c4bd911c\") " Sep 10 15:04:02.500538 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500345 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-data" (OuterVolumeSpecName: "istio-data") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:02.500538 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500413 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:02.500538 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500438 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:04:02.500538 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500332 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:02.500828 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500784 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.500882 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500835 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.500882 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500850 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.500882 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500863 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.500882 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.500876 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.502181 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.502154 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-kube-api-access-mfs8s" (OuterVolumeSpecName: "kube-api-access-mfs8s") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "kube-api-access-mfs8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:04:02.502311 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.502290 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-token" (OuterVolumeSpecName: "istio-token") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:04:02.502704 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.502678 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:02.502704 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.502693 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "10dc16fc-089f-447a-bfa3-3bc6c4bd911c" (UID: "10dc16fc-089f-447a-bfa3-3bc6c4bd911c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:04:02.602119 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.602087 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfs8s\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-kube-api-access-mfs8s\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.602119 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.602114 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.602119 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.602123 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.602340 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.602131 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10dc16fc-089f-447a-bfa3-3bc6c4bd911c-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:02.760890 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.760856 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln"] Sep 10 15:04:02.765040 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:02.765016 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c3f76b1a-openshift-default-5fb56bd4dd-gwwln"] Sep 10 15:04:03.664987 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:03.664960 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" path="/var/lib/kubelet/pods/10dc16fc-089f-447a-bfa3-3bc6c4bd911c/volumes" Sep 10 15:04:19.167312 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.167281 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx"] Sep 10 15:04:19.167758 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.167555 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" containerID="cri-o://f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb" gracePeriod=30 Sep 10 15:04:19.900774 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.900739 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r"] Sep 10 15:04:19.901394 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901372 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" Sep 10 15:04:19.901470 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901398 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" Sep 10 15:04:19.901470 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901410 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" Sep 10 15:04:19.901470 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901419 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" Sep 10 15:04:19.901470 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901440 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" Sep 10 15:04:19.901470 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901449 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" Sep 10 15:04:19.901742 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901472 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542fe8bc-53f3-4020-a112-e5753f00aced" containerName="maas-api" Sep 10 15:04:19.901742 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901481 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="542fe8bc-53f3-4020-a112-e5753f00aced" containerName="maas-api" Sep 10 15:04:19.901742 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901584 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" Sep 10 15:04:19.901742 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901599 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="10dc16fc-089f-447a-bfa3-3bc6c4bd911c" containerName="istio-proxy" Sep 10 15:04:19.901742 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901610 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" Sep 10 15:04:19.901742 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901620 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="542fe8bc-53f3-4020-a112-e5753f00aced" containerName="maas-api" Sep 10 15:04:19.901742 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.901629 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" Sep 10 15:04:19.904837 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.904814 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.907398 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.907377 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-78f25d-openshift-default-dockercfg-cdmsb\"" Sep 10 15:04:19.921496 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.921470 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r"] Sep 10 15:04:19.966382 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966354 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-data\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.966511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966417 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-envoy\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.966511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966446 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-credential-socket\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.966511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966464 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-socket\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.966614 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966526 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-kube-api-access-dct99\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.966614 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966559 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-certs\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.966614 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966587 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-istio-token\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.966707 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966621 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234aba30-9757-49cb-9e79-ea684a7883f4-istio-podinfo\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:19.966707 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:19.966666 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234aba30-9757-49cb-9e79-ea684a7883f4-istiod-ca-cert\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068138 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068101 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-kube-api-access-dct99\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068138 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-certs\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068373 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068170 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-istio-token\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068373 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068211 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234aba30-9757-49cb-9e79-ea684a7883f4-istio-podinfo\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068373 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068266 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234aba30-9757-49cb-9e79-ea684a7883f4-istiod-ca-cert\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068373 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068324 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-data\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068582 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068392 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-envoy\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068582 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-credential-socket\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068582 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068458 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-socket\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068730 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068581 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-certs\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068837 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068793 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-data\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068916 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068895 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-socket\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.068962 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.068932 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-credential-socket\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.069285 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.069261 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234aba30-9757-49cb-9e79-ea684a7883f4-istiod-ca-cert\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.070734 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.070711 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-envoy\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.070967 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.070944 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234aba30-9757-49cb-9e79-ea684a7883f4-istio-podinfo\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.076932 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.076906 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-istio-token\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.077135 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.077116 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-kube-api-access-dct99\") pod \"e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.217029 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.216953 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:20.499258 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.499232 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r"] Sep 10 15:04:20.500774 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:04:20.500744 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234aba30_9757_49cb_9e79_ea684a7883f4.slice/crio-a8bc48d6b88b0614f172333eca431eb593eebf4d201af13125bb11b416c9d20f WatchSource:0}: Error finding container a8bc48d6b88b0614f172333eca431eb593eebf4d201af13125bb11b416c9d20f: Status 404 returned error can't find the container with id a8bc48d6b88b0614f172333eca431eb593eebf4d201af13125bb11b416c9d20f Sep 10 15:04:20.503467 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.503434 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:04:20.503571 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.503513 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:04:20.503571 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.503559 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:04:20.532555 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:20.532533 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" event={"ID":"234aba30-9757-49cb-9e79-ea684a7883f4","Type":"ContainerStarted","Data":"a8bc48d6b88b0614f172333eca431eb593eebf4d201af13125bb11b416c9d20f"} Sep 10 15:04:21.539081 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:21.539048 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" event={"ID":"234aba30-9757-49cb-9e79-ea684a7883f4","Type":"ContainerStarted","Data":"9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15"} Sep 10 15:04:21.563513 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:21.563464 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" podStartSLOduration=2.5634457 podStartE2EDuration="2.5634457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:04:21.561487634 +0000 UTC m=+1000.376070386" watchObservedRunningTime="2026-09-10 15:04:21.5634457 +0000 UTC m=+1000.378028443" Sep 10 15:04:22.217602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:22.217567 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:22.222460 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:22.222434 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:22.544396 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:22.544320 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:22.545291 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:22.545268 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:24.416970 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.416942 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:04:24.510569 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510494 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-envoy\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510709 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510573 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-data\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510709 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510592 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-certs\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510709 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510636 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-socket\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510709 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510656 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-credential-socket\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510709 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510682 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbwrz\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-kube-api-access-zbwrz\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510996 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510722 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb72069-209b-4ce9-bdd9-55f44f034466-istio-podinfo\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510996 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510750 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb72069-209b-4ce9-bdd9-55f44f034466-istiod-ca-cert\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510996 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510838 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-istio-token\") pod \"abb72069-209b-4ce9-bdd9-55f44f034466\" (UID: \"abb72069-209b-4ce9-bdd9-55f44f034466\") " Sep 10 15:04:24.510996 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510890 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:24.510996 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.510948 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-data" (OuterVolumeSpecName: "istio-data") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:24.511284 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.511018 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:24.511284 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.511137 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:24.511284 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.511249 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb72069-209b-4ce9-bdd9-55f44f034466-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:04:24.511446 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.511332 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.511446 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.511352 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.511446 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.511365 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.511446 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.511378 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.512728 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.512699 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:04:24.512947 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.512915 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abb72069-209b-4ce9-bdd9-55f44f034466-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:04:24.513063 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.513044 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-istio-token" (OuterVolumeSpecName: "istio-token") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:04:24.513110 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.513073 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-kube-api-access-zbwrz" (OuterVolumeSpecName: "kube-api-access-zbwrz") pod "abb72069-209b-4ce9-bdd9-55f44f034466" (UID: "abb72069-209b-4ce9-bdd9-55f44f034466"). InnerVolumeSpecName "kube-api-access-zbwrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:04:24.557395 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.557357 2560 generic.go:356] "Generic (PLEG): container finished" podID="abb72069-209b-4ce9-bdd9-55f44f034466" containerID="f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb" exitCode=0 Sep 10 15:04:24.557502 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.557410 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" Sep 10 15:04:24.557502 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.557409 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" event={"ID":"abb72069-209b-4ce9-bdd9-55f44f034466","Type":"ContainerDied","Data":"f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb"} Sep 10 15:04:24.557502 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.557473 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx" event={"ID":"abb72069-209b-4ce9-bdd9-55f44f034466","Type":"ContainerDied","Data":"b919780dc2ae82ab46605789369ed90669bb28fb20b17109bb593e5950267b19"} Sep 10 15:04:24.557615 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.557512 2560 scope.go:117] "RemoveContainer" containerID="f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb" Sep 10 15:04:24.566272 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.566256 2560 scope.go:117] "RemoveContainer" containerID="68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df" Sep 10 15:04:24.573855 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.573835 2560 scope.go:117] "RemoveContainer" containerID="f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb" Sep 10 15:04:24.574124 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:04:24.574103 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb\": container with ID starting with f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb not found: ID does not exist" containerID="f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb" Sep 10 15:04:24.574193 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.574130 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb"} err="failed to get container status \"f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb\": rpc error: code = NotFound desc = could not find container \"f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb\": container with ID starting with f604f6bfe174a84ea2b0f32ff05831381bd05c4e962afc57b7621252d93b2afb not found: ID does not exist" Sep 10 15:04:24.574193 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.574146 2560 scope.go:117] "RemoveContainer" containerID="68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df" Sep 10 15:04:24.574386 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:04:24.574369 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df\": container with ID starting with 68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df not found: ID does not exist" containerID="68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df" Sep 10 15:04:24.574428 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.574392 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df"} err="failed to get container status \"68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df\": rpc error: code = NotFound desc = could not find container \"68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df\": container with ID starting with 68d4b78b620cbc8e2d2151ea5c1b50a29fd6656ea322317de2d09b6ba59b22df not found: ID does not exist" Sep 10 15:04:24.581015 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.580992 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx"] Sep 10 15:04:24.585413 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.585393 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c75e68-openshift-default-59d978c98-ckkfx"] Sep 10 15:04:24.612392 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.612374 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbwrz\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-kube-api-access-zbwrz\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.612475 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.612396 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abb72069-209b-4ce9-bdd9-55f44f034466-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.612475 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.612406 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abb72069-209b-4ce9-bdd9-55f44f034466-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.612475 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.612414 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abb72069-209b-4ce9-bdd9-55f44f034466-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.612475 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.612422 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abb72069-209b-4ce9-bdd9-55f44f034466-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:04:24.715438 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.715409 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8"] Sep 10 15:04:24.716060 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.716041 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" Sep 10 15:04:24.716114 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.716063 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" Sep 10 15:04:24.716114 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.716083 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" Sep 10 15:04:24.716114 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.716092 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd0fd4b-9de1-463f-94ac-efbf1919a28e" containerName="istio-proxy" Sep 10 15:04:24.716209 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.716195 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" Sep 10 15:04:24.721144 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.721103 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.723614 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.723588 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-78f25d-serving-cert\"" Sep 10 15:04:24.735037 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.735015 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8"] Sep 10 15:04:24.814842 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.814082 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-metrics-certs\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.814842 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.814162 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.814842 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.814231 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5b2987-710b-4f6c-8f09-6533513214d2-openshift-service-ca\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.814842 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.814278 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6xt\" (UniqueName: \"kubernetes.io/projected/1b5b2987-710b-4f6c-8f09-6533513214d2-kube-api-access-rb6xt\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.915637 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.915604 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.915853 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.915666 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5b2987-710b-4f6c-8f09-6533513214d2-openshift-service-ca\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.915853 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.915707 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6xt\" (UniqueName: \"kubernetes.io/projected/1b5b2987-710b-4f6c-8f09-6533513214d2-kube-api-access-rb6xt\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.915853 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:04:24.915753 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-serving-cert: secret "maas-api-e2e-explicit-78f25d-serving-cert" not found Sep 10 15:04:24.915853 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.915831 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-metrics-certs\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.915853 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:04:24.915856 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls podName:1b5b2987-710b-4f6c-8f09-6533513214d2 nodeName:}" failed. No retries permitted until 2026-09-10 15:04:25.415835387 +0000 UTC m=+1004.230418133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls") pod "maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" (UID: "1b5b2987-710b-4f6c-8f09-6533513214d2") : secret "maas-api-e2e-explicit-78f25d-serving-cert" not found Sep 10 15:04:24.916424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.916401 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5b2987-710b-4f6c-8f09-6533513214d2-openshift-service-ca\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.918135 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.918107 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-metrics-certs\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:24.933033 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:24.933015 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6xt\" (UniqueName: \"kubernetes.io/projected/1b5b2987-710b-4f6c-8f09-6533513214d2-kube-api-access-rb6xt\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:25.420584 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:25.420548 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:25.422881 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:25.422858 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls\") pod \"maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:25.631982 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:25.631942 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:25.665122 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:25.665092 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" path="/var/lib/kubelet/pods/abb72069-209b-4ce9-bdd9-55f44f034466/volumes" Sep 10 15:04:25.908597 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:25.908575 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8"] Sep 10 15:04:25.909871 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:04:25.909848 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5b2987_710b_4f6c_8f09_6533513214d2.slice/crio-c7e02517ba1fbaa4723ca64021e09e00dae09e42ef1b5d914dddc94a27062d6b WatchSource:0}: Error finding container c7e02517ba1fbaa4723ca64021e09e00dae09e42ef1b5d914dddc94a27062d6b: Status 404 returned error can't find the container with id c7e02517ba1fbaa4723ca64021e09e00dae09e42ef1b5d914dddc94a27062d6b Sep 10 15:04:26.570371 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:26.570332 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" event={"ID":"1b5b2987-710b-4f6c-8f09-6533513214d2","Type":"ContainerStarted","Data":"c7e02517ba1fbaa4723ca64021e09e00dae09e42ef1b5d914dddc94a27062d6b"} Sep 10 15:04:27.577110 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:27.577077 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" event={"ID":"1b5b2987-710b-4f6c-8f09-6533513214d2","Type":"ContainerStarted","Data":"91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d"} Sep 10 15:04:27.577481 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:27.577152 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:27.597239 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:27.597190 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" podStartSLOduration=2.5101623809999998 podStartE2EDuration="3.597175557s" totalImagesPullingTime="1.087013176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:04:27.595595663 +0000 UTC m=+1006.410178419" watchObservedRunningTime="2026-09-10 15:04:27.597175557 +0000 UTC m=+1006.411758307" Sep 10 15:04:33.588151 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:33.588121 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:04:44.672878 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:44.672835 2560 generic.go:356] "Generic (PLEG): container finished" podID="234aba30-9757-49cb-9e79-ea684a7883f4" containerID="9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15" exitCode=0 Sep 10 15:04:44.673394 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:44.672922 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" event={"ID":"234aba30-9757-49cb-9e79-ea684a7883f4","Type":"ContainerDied","Data":"9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15"} Sep 10 15:04:44.673475 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:44.673419 2560 scope.go:117] "RemoveContainer" containerID="9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15" Sep 10 15:04:44.674591 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:44.674562 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:04:44.674656 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:44.674641 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:04:44.674693 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:44.674680 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:04:45.684041 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:45.683999 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" event={"ID":"234aba30-9757-49cb-9e79-ea684a7883f4","Type":"ContainerStarted","Data":"0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465"} Sep 10 15:04:46.217755 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:46.217724 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:46.217953 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:46.217931 2560 patch_prober.go:28] interesting pod/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:04:46.218001 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:46.217978 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 10 15:04:47.218178 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:47.218145 2560 patch_prober.go:28] interesting pod/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:04:47.218573 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:47.218215 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 10 15:04:48.218192 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:48.218157 2560 patch_prober.go:28] interesting pod/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:04:48.218565 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:48.218217 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 10 15:04:49.217449 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:49.217416 2560 patch_prober.go:28] interesting pod/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:04:49.217616 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:49.217467 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 10 15:04:50.217306 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:50.217268 2560 patch_prober.go:28] interesting pod/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:04:50.217676 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:50.217323 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 10 15:04:51.366209 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:51.366178 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:51.366731 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:51.366705 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:04:51.367713 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:04:51.367695 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:05:18.346027 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.345987 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8"] Sep 10 15:05:18.346591 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.346258 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" podUID="1b5b2987-710b-4f6c-8f09-6533513214d2" containerName="maas-api" containerID="cri-o://91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d" gracePeriod=30 Sep 10 15:05:18.589615 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.589593 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:05:18.754493 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.754455 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls\") pod \"1b5b2987-710b-4f6c-8f09-6533513214d2\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " Sep 10 15:05:18.754715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.754518 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-metrics-certs\") pod \"1b5b2987-710b-4f6c-8f09-6533513214d2\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " Sep 10 15:05:18.754715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.754560 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5b2987-710b-4f6c-8f09-6533513214d2-openshift-service-ca\") pod \"1b5b2987-710b-4f6c-8f09-6533513214d2\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " Sep 10 15:05:18.754715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.754587 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb6xt\" (UniqueName: \"kubernetes.io/projected/1b5b2987-710b-4f6c-8f09-6533513214d2-kube-api-access-rb6xt\") pod \"1b5b2987-710b-4f6c-8f09-6533513214d2\" (UID: \"1b5b2987-710b-4f6c-8f09-6533513214d2\") " Sep 10 15:05:18.755297 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.755266 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5b2987-710b-4f6c-8f09-6533513214d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b5b2987-710b-4f6c-8f09-6533513214d2" (UID: "1b5b2987-710b-4f6c-8f09-6533513214d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:18.756992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.756965 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b5b2987-710b-4f6c-8f09-6533513214d2" (UID: "1b5b2987-710b-4f6c-8f09-6533513214d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:05:18.757303 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.757266 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1b5b2987-710b-4f6c-8f09-6533513214d2" (UID: "1b5b2987-710b-4f6c-8f09-6533513214d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:05:18.757388 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.757347 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5b2987-710b-4f6c-8f09-6533513214d2-kube-api-access-rb6xt" (OuterVolumeSpecName: "kube-api-access-rb6xt") pod "1b5b2987-710b-4f6c-8f09-6533513214d2" (UID: "1b5b2987-710b-4f6c-8f09-6533513214d2"). InnerVolumeSpecName "kube-api-access-rb6xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:18.853711 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.853675 2560 generic.go:356] "Generic (PLEG): container finished" podID="1b5b2987-710b-4f6c-8f09-6533513214d2" containerID="91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d" exitCode=0 Sep 10 15:05:18.853895 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.853719 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" Sep 10 15:05:18.853895 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.853759 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" event={"ID":"1b5b2987-710b-4f6c-8f09-6533513214d2","Type":"ContainerDied","Data":"91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d"} Sep 10 15:05:18.853895 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.853838 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" event={"ID":"1b5b2987-710b-4f6c-8f09-6533513214d2","Type":"ContainerDied","Data":"c7e02517ba1fbaa4723ca64021e09e00dae09e42ef1b5d914dddc94a27062d6b"} Sep 10 15:05:18.853895 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.853858 2560 scope.go:117] "RemoveContainer" containerID="91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d" Sep 10 15:05:18.855521 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.855505 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:18.855521 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.855522 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5b2987-710b-4f6c-8f09-6533513214d2-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:18.855646 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.855531 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6xt\" (UniqueName: \"kubernetes.io/projected/1b5b2987-710b-4f6c-8f09-6533513214d2-kube-api-access-rb6xt\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:18.855646 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.855541 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5b2987-710b-4f6c-8f09-6533513214d2-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:18.863666 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.863650 2560 scope.go:117] "RemoveContainer" containerID="91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d" Sep 10 15:05:18.863948 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:05:18.863931 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d\": container with ID starting with 91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d not found: ID does not exist" containerID="91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d" Sep 10 15:05:18.864026 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.863954 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d"} err="failed to get container status \"91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d\": rpc error: code = NotFound desc = could not find container \"91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d\": container with ID starting with 91b0cd305c718627c553776f347eb7c30664a6aab336e98dcba47c72502ee02d not found: ID does not exist" Sep 10 15:05:18.878264 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.878241 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8"] Sep 10 15:05:18.882794 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:18.882775 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8"] Sep 10 15:05:19.665048 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:19.665011 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b5b2987-710b-4f6c-8f09-6533513214d2" path="/var/lib/kubelet/pods/1b5b2987-710b-4f6c-8f09-6533513214d2/volumes" Sep 10 15:05:21.583420 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:21.583371 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78f25d-6488974fcc-7hlc8" podUID="1b5b2987-710b-4f6c-8f09-6533513214d2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.44:8443/health\": context deadline exceeded" Sep 10 15:05:25.593590 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:25.593549 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r"] Sep 10 15:05:25.594083 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:25.593775 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" containerID="cri-o://0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465" gracePeriod=30 Sep 10 15:05:26.355302 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.355266 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r"] Sep 10 15:05:26.355965 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.355942 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" Sep 10 15:05:26.355965 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.355966 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" Sep 10 15:05:26.356142 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.355996 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b5b2987-710b-4f6c-8f09-6533513214d2" containerName="maas-api" Sep 10 15:05:26.356142 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.356005 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b5b2987-710b-4f6c-8f09-6533513214d2" containerName="maas-api" Sep 10 15:05:26.356142 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.356109 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb72069-209b-4ce9-bdd9-55f44f034466" containerName="istio-proxy" Sep 10 15:05:26.356142 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.356132 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b5b2987-710b-4f6c-8f09-6533513214d2" containerName="maas-api" Sep 10 15:05:26.359593 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.359572 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.363580 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.363554 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-328f99-openshift-default-dockercfg-kq5br\"" Sep 10 15:05:26.373279 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.373258 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r"] Sep 10 15:05:26.420564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420535 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be72303-e148-468e-949e-f9f58f8dc6c2-istiod-ca-cert\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.420776 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420599 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-credential-socket\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.420776 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420629 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-envoy\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.420776 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420666 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-socket\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.420776 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420724 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzvq\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-kube-api-access-7jzvq\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.420776 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420769 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-token\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.421029 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420812 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-certs\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.421029 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420930 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-data\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.421029 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.420966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-podinfo\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522299 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522252 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-token\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522506 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522321 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-certs\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522506 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-data\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522506 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522438 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-podinfo\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522692 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522519 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be72303-e148-468e-949e-f9f58f8dc6c2-istiod-ca-cert\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522692 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522586 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-credential-socket\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522692 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522620 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-envoy\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522692 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522671 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-socket\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.522949 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.522706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzvq\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-kube-api-access-7jzvq\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.523832 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.523785 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-certs\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.524076 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.524057 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-data\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.524763 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.524738 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-credential-socket\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.526700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.524857 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-socket\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.526700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.525481 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be72303-e148-468e-949e-f9f58f8dc6c2-istiod-ca-cert\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.530084 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.528888 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-envoy\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.530084 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.529200 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-podinfo\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.532912 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.532711 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-token\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.533791 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.533772 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzvq\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-kube-api-access-7jzvq\") pod \"e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.674848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.674814 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:26.981778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.981751 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r"] Sep 10 15:05:26.982733 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:05:26.982707 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be72303_e148_468e_949e_f9f58f8dc6c2.slice/crio-e98e231c9a89ae47eaf1bbbc3bc008f992258d2f66f25618df2b9d0c6738d433 WatchSource:0}: Error finding container e98e231c9a89ae47eaf1bbbc3bc008f992258d2f66f25618df2b9d0c6738d433: Status 404 returned error can't find the container with id e98e231c9a89ae47eaf1bbbc3bc008f992258d2f66f25618df2b9d0c6738d433 Sep 10 15:05:26.985047 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.985012 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:05:26.985133 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.985078 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:05:26.985133 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:26.985110 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:05:27.899660 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:27.899625 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" event={"ID":"3be72303-e148-468e-949e-f9f58f8dc6c2","Type":"ContainerStarted","Data":"1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085"} Sep 10 15:05:27.900085 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:27.899678 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" event={"ID":"3be72303-e148-468e-949e-f9f58f8dc6c2","Type":"ContainerStarted","Data":"e98e231c9a89ae47eaf1bbbc3bc008f992258d2f66f25618df2b9d0c6738d433"} Sep 10 15:05:27.926709 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:27.926663 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" podStartSLOduration=1.926647874 podStartE2EDuration="1.926647874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:05:27.924559175 +0000 UTC m=+1066.739141927" watchObservedRunningTime="2026-09-10 15:05:27.926647874 +0000 UTC m=+1066.741230624" Sep 10 15:05:28.675143 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:28.675111 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:28.680208 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:28.680181 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:28.905236 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:28.905203 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:28.906153 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:28.906134 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:30.843984 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.843759 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:05:30.867272 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867253 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-envoy\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867406 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867296 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-socket\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867406 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867314 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234aba30-9757-49cb-9e79-ea684a7883f4-istio-podinfo\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867406 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867342 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234aba30-9757-49cb-9e79-ea684a7883f4-istiod-ca-cert\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867406 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867363 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-credential-socket\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867630 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867484 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-data\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867630 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867525 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:30.867630 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867559 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-istio-token\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867630 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867591 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-certs\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867894 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867644 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:30.867894 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867705 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-kube-api-access-dct99\") pod \"234aba30-9757-49cb-9e79-ea684a7883f4\" (UID: \"234aba30-9757-49cb-9e79-ea684a7883f4\") " Sep 10 15:05:30.867894 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867765 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:30.868057 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867837 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234aba30-9757-49cb-9e79-ea684a7883f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:30.868057 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.867889 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:30.868153 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.868098 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:30.868153 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.868112 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:30.868153 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.868123 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234aba30-9757-49cb-9e79-ea684a7883f4-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:30.868153 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.868136 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:30.868153 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.868144 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:30.869535 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.869513 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/234aba30-9757-49cb-9e79-ea684a7883f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:05:30.869722 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.869703 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-kube-api-access-dct99" (OuterVolumeSpecName: "kube-api-access-dct99") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "kube-api-access-dct99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:30.869878 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.869859 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:30.869929 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.869877 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "234aba30-9757-49cb-9e79-ea684a7883f4" (UID: "234aba30-9757-49cb-9e79-ea684a7883f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:30.917026 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.916996 2560 generic.go:356] "Generic (PLEG): container finished" podID="234aba30-9757-49cb-9e79-ea684a7883f4" containerID="0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465" exitCode=0 Sep 10 15:05:30.917194 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.917032 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" event={"ID":"234aba30-9757-49cb-9e79-ea684a7883f4","Type":"ContainerDied","Data":"0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465"} Sep 10 15:05:30.917194 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.917086 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" event={"ID":"234aba30-9757-49cb-9e79-ea684a7883f4","Type":"ContainerDied","Data":"a8bc48d6b88b0614f172333eca431eb593eebf4d201af13125bb11b416c9d20f"} Sep 10 15:05:30.917194 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.917047 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r" Sep 10 15:05:30.917194 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.917107 2560 scope.go:117] "RemoveContainer" containerID="0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465" Sep 10 15:05:30.926302 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.926285 2560 scope.go:117] "RemoveContainer" containerID="9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15" Sep 10 15:05:30.934129 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.934110 2560 scope.go:117] "RemoveContainer" containerID="0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465" Sep 10 15:05:30.934395 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:05:30.934375 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465\": container with ID starting with 0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465 not found: ID does not exist" containerID="0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465" Sep 10 15:05:30.934458 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.934403 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465"} err="failed to get container status \"0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465\": rpc error: code = NotFound desc = could not find container \"0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465\": container with ID starting with 0a0e41e04ab20b2e6a691a866430e2c18520378d6805b79bfb35a5116d01b465 not found: ID does not exist" Sep 10 15:05:30.934458 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.934418 2560 scope.go:117] "RemoveContainer" containerID="9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15" Sep 10 15:05:30.934678 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:05:30.934662 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15\": container with ID starting with 9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15 not found: ID does not exist" containerID="9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15" Sep 10 15:05:30.934722 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.934684 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15"} err="failed to get container status \"9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15\": rpc error: code = NotFound desc = could not find container \"9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15\": container with ID starting with 9632688592b9fbb6ec7284865aa4318b61ac86186fb5729ebda701f88e69fb15 not found: ID does not exist" Sep 10 15:05:30.949824 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.947241 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r"] Sep 10 15:05:30.953191 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.953166 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-78f25d-openshift-default-5874bfd596-rq52r"] Sep 10 15:05:30.968970 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.968945 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dct99\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-kube-api-access-dct99\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:30.968970 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.968966 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234aba30-9757-49cb-9e79-ea684a7883f4-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:30.969098 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.968977 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234aba30-9757-49cb-9e79-ea684a7883f4-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:30.969098 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:30.968984 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234aba30-9757-49cb-9e79-ea684a7883f4-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:31.664591 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:31.664558 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" path="/var/lib/kubelet/pods/234aba30-9757-49cb-9e79-ea684a7883f4/volumes" Sep 10 15:05:45.537132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.537100 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9"] Sep 10 15:05:45.537543 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.537528 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" Sep 10 15:05:45.537543 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.537544 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" Sep 10 15:05:45.537629 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.537619 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" Sep 10 15:05:45.537662 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.537629 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" Sep 10 15:05:45.542118 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.542095 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.544794 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.544774 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-411a8086-openshift-default-dockercfg-67ds2\"" Sep 10 15:05:45.558395 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.558371 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9"] Sep 10 15:05:45.606414 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606389 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-podinfo\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.606557 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606425 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-credential-socket\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.606557 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606461 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-envoy\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.606685 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606563 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4k9\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-kube-api-access-kv4k9\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.606685 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606605 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-data\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.606775 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606681 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-token\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.606775 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606704 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-certs\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.606775 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606734 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-socket\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.606938 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.606826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d35a27-7fee-4255-b582-ded92b48cd2f-istiod-ca-cert\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.707990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.707955 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d35a27-7fee-4255-b582-ded92b48cd2f-istiod-ca-cert\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708165 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708042 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-podinfo\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708165 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708075 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-credential-socket\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708165 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708132 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-envoy\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708363 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708167 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4k9\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-kube-api-access-kv4k9\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708363 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708195 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-data\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708363 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708232 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-token\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708363 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708259 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-certs\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708579 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708485 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-socket\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708694 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708669 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-data\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708891 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708784 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-certs\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708891 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708836 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-credential-socket\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.708891 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.708871 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-socket\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.709082 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.709020 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d35a27-7fee-4255-b582-ded92b48cd2f-istiod-ca-cert\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.710625 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.710594 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-podinfo\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.710747 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.710681 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-envoy\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.719970 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.719946 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-token\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.720217 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.720193 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4k9\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-kube-api-access-kv4k9\") pod \"e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:45.852239 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:45.852149 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:46.130151 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:46.130124 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9"] Sep 10 15:05:46.132282 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:05:46.132258 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d35a27_7fee_4255_b582_ded92b48cd2f.slice/crio-9e289a535eb51fe3ff75f8c871c486d414e9440eabb392d16ffdf64482a28c47 WatchSource:0}: Error finding container 9e289a535eb51fe3ff75f8c871c486d414e9440eabb392d16ffdf64482a28c47: Status 404 returned error can't find the container with id 9e289a535eb51fe3ff75f8c871c486d414e9440eabb392d16ffdf64482a28c47 Sep 10 15:05:46.134577 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:46.134540 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:05:46.134677 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:46.134618 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:05:46.134737 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:46.134691 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:05:46.672243 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:46.672214 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r"] Sep 10 15:05:46.672579 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:46.672397 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" podUID="3be72303-e148-468e-949e-f9f58f8dc6c2" containerName="istio-proxy" containerID="cri-o://1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085" gracePeriod=30 Sep 10 15:05:46.995624 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:46.995538 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" event={"ID":"39d35a27-7fee-4255-b582-ded92b48cd2f","Type":"ContainerStarted","Data":"8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13"} Sep 10 15:05:46.995624 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:46.995597 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" event={"ID":"39d35a27-7fee-4255-b582-ded92b48cd2f","Type":"ContainerStarted","Data":"9e289a535eb51fe3ff75f8c871c486d414e9440eabb392d16ffdf64482a28c47"} Sep 10 15:05:47.019511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:47.019465 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" podStartSLOduration=2.019448781 podStartE2EDuration="2.019448781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:05:47.01746089 +0000 UTC m=+1085.832043642" watchObservedRunningTime="2026-09-10 15:05:47.019448781 +0000 UTC m=+1085.834031531" Sep 10 15:05:47.853336 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:47.853303 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:47.858061 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:47.858037 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:48.000225 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:48.000191 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:48.001032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:48.001013 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:05:50.097043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.097002 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v"] Sep 10 15:05:50.097627 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.097606 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" Sep 10 15:05:50.097627 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.097624 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="234aba30-9757-49cb-9e79-ea684a7883f4" containerName="istio-proxy" Sep 10 15:05:50.101061 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.101038 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.103273 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.103253 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-411a8086-serving-cert\"" Sep 10 15:05:50.112828 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.112809 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v"] Sep 10 15:05:50.257462 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.257433 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-maas-api-tls\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.257632 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.257477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl6lq\" (UniqueName: \"kubernetes.io/projected/4e4229cb-b62e-4f38-b010-386602882887-kube-api-access-pl6lq\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.257632 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.257500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4229cb-b62e-4f38-b010-386602882887-openshift-service-ca\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.257632 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.257523 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-metrics-certs\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.359129 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.359042 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-maas-api-tls\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.359129 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.359087 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6lq\" (UniqueName: \"kubernetes.io/projected/4e4229cb-b62e-4f38-b010-386602882887-kube-api-access-pl6lq\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.359129 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.359113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4229cb-b62e-4f38-b010-386602882887-openshift-service-ca\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.359129 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.359127 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-metrics-certs\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.359708 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.359687 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4229cb-b62e-4f38-b010-386602882887-openshift-service-ca\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.361510 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.361484 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-maas-api-tls\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.361597 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.361537 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-metrics-certs\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.367717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.367696 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6lq\" (UniqueName: \"kubernetes.io/projected/4e4229cb-b62e-4f38-b010-386602882887-kube-api-access-pl6lq\") pod \"maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.413896 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.413871 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:50.685883 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:50.685855 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v"] Sep 10 15:05:50.688219 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:05:50.688192 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4229cb_b62e_4f38_b010_386602882887.slice/crio-38c1f55c4204d675ace83a815d74f3764144288135140f6db6c228406cb1ce48 WatchSource:0}: Error finding container 38c1f55c4204d675ace83a815d74f3764144288135140f6db6c228406cb1ce48: Status 404 returned error can't find the container with id 38c1f55c4204d675ace83a815d74f3764144288135140f6db6c228406cb1ce48 Sep 10 15:05:51.017191 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:51.017106 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" event={"ID":"4e4229cb-b62e-4f38-b010-386602882887","Type":"ContainerStarted","Data":"38c1f55c4204d675ace83a815d74f3764144288135140f6db6c228406cb1ce48"} Sep 10 15:05:51.925614 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:51.925593 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:52.024226 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.024187 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" event={"ID":"4e4229cb-b62e-4f38-b010-386602882887","Type":"ContainerStarted","Data":"647a9a6f7dab6923e34ac57ee1c221a7a23ced229922e8e13153cdc409d050b2"} Sep 10 15:05:52.024436 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.024256 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:52.025460 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.025434 2560 generic.go:356] "Generic (PLEG): container finished" podID="3be72303-e148-468e-949e-f9f58f8dc6c2" containerID="1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085" exitCode=0 Sep 10 15:05:52.025567 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.025478 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" Sep 10 15:05:52.025567 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.025517 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" event={"ID":"3be72303-e148-468e-949e-f9f58f8dc6c2","Type":"ContainerDied","Data":"1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085"} Sep 10 15:05:52.025567 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.025564 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r" event={"ID":"3be72303-e148-468e-949e-f9f58f8dc6c2","Type":"ContainerDied","Data":"e98e231c9a89ae47eaf1bbbc3bc008f992258d2f66f25618df2b9d0c6738d433"} Sep 10 15:05:52.025677 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.025588 2560 scope.go:117] "RemoveContainer" containerID="1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085" Sep 10 15:05:52.034361 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.034343 2560 scope.go:117] "RemoveContainer" containerID="1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085" Sep 10 15:05:52.034623 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:05:52.034603 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085\": container with ID starting with 1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085 not found: ID does not exist" containerID="1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085" Sep 10 15:05:52.034720 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.034628 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085"} err="failed to get container status \"1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085\": rpc error: code = NotFound desc = could not find container \"1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085\": container with ID starting with 1b4f2a3cbc6cef26b9440ee00053b20bcff748686d54b8fe87f6f5da410c1085 not found: ID does not exist" Sep 10 15:05:52.041175 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.041135 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" podStartSLOduration=1.167670378 podStartE2EDuration="2.041122399s" totalImagesPullingTime="873.452021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:05:52.040951578 +0000 UTC m=+1090.855534331" watchObservedRunningTime="2026-09-10 15:05:52.041122399 +0000 UTC m=+1090.855705163" Sep 10 15:05:52.079346 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079323 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-certs\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079449 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079354 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-podinfo\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079449 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079372 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-credential-socket\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079449 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079398 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-data\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079449 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079427 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be72303-e148-468e-949e-f9f58f8dc6c2-istiod-ca-cert\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079668 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079475 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-token\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079668 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079525 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jzvq\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-kube-api-access-7jzvq\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079668 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079599 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-envoy\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079668 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079603 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:52.079668 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079662 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-socket\") pod \"3be72303-e148-468e-949e-f9f58f8dc6c2\" (UID: \"3be72303-e148-468e-949e-f9f58f8dc6c2\") " Sep 10 15:05:52.079963 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079684 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:52.079963 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.079702 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:52.080064 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.080001 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:52.080064 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.080025 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be72303-e148-468e-949e-f9f58f8dc6c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:52.080173 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.080101 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.080173 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.080118 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.080173 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.080131 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.080173 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.080144 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.082202 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.082166 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-kube-api-access-7jzvq" (OuterVolumeSpecName: "kube-api-access-7jzvq") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "kube-api-access-7jzvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:52.082308 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.082240 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:05:52.082308 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.082245 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:05:52.082308 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.082269 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "3be72303-e148-468e-949e-f9f58f8dc6c2" (UID: "3be72303-e148-468e-949e-f9f58f8dc6c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:52.181778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.181660 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.181778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.181692 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.181778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.181707 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be72303-e148-468e-949e-f9f58f8dc6c2-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.181778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.181719 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.181778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.181732 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzvq\" (UniqueName: \"kubernetes.io/projected/3be72303-e148-468e-949e-f9f58f8dc6c2-kube-api-access-7jzvq\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:52.349939 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.349909 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r"] Sep 10 15:05:52.353474 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:52.353450 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-328f99-openshift-default-7dbdff8b6-45h8r"] Sep 10 15:05:53.665529 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:53.665494 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3be72303-e148-468e-949e-f9f58f8dc6c2" path="/var/lib/kubelet/pods/3be72303-e148-468e-949e-f9f58f8dc6c2/volumes" Sep 10 15:05:58.040508 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:58.040480 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:05:58.593396 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:58.593358 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr"] Sep 10 15:05:58.593677 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:58.593647 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" podUID="f8466e73-ba2c-474b-892b-a2ca11bb9fad" containerName="maas-api" containerID="cri-o://d6090e1121ba90e8d840bb07cca8a720ec1aa7ec7edcea2312375c8342080729" gracePeriod=30 Sep 10 15:05:59.082624 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.082583 2560 generic.go:356] "Generic (PLEG): container finished" podID="f8466e73-ba2c-474b-892b-a2ca11bb9fad" containerID="d6090e1121ba90e8d840bb07cca8a720ec1aa7ec7edcea2312375c8342080729" exitCode=0 Sep 10 15:05:59.083046 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.082645 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" event={"ID":"f8466e73-ba2c-474b-892b-a2ca11bb9fad","Type":"ContainerDied","Data":"d6090e1121ba90e8d840bb07cca8a720ec1aa7ec7edcea2312375c8342080729"} Sep 10 15:05:59.103772 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.103752 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:05:59.245992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.245907 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-metrics-certs\") pod \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " Sep 10 15:05:59.246145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.245992 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zndnv\" (UniqueName: \"kubernetes.io/projected/f8466e73-ba2c-474b-892b-a2ca11bb9fad-kube-api-access-zndnv\") pod \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " Sep 10 15:05:59.246145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.246026 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8466e73-ba2c-474b-892b-a2ca11bb9fad-openshift-service-ca\") pod \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " Sep 10 15:05:59.246145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.246048 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls\") pod \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\" (UID: \"f8466e73-ba2c-474b-892b-a2ca11bb9fad\") " Sep 10 15:05:59.246401 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.246368 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8466e73-ba2c-474b-892b-a2ca11bb9fad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8466e73-ba2c-474b-892b-a2ca11bb9fad" (UID: "f8466e73-ba2c-474b-892b-a2ca11bb9fad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:05:59.248211 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.248182 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8466e73-ba2c-474b-892b-a2ca11bb9fad" (UID: "f8466e73-ba2c-474b-892b-a2ca11bb9fad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:05:59.248211 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.248199 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8466e73-ba2c-474b-892b-a2ca11bb9fad-kube-api-access-zndnv" (OuterVolumeSpecName: "kube-api-access-zndnv") pod "f8466e73-ba2c-474b-892b-a2ca11bb9fad" (UID: "f8466e73-ba2c-474b-892b-a2ca11bb9fad"). InnerVolumeSpecName "kube-api-access-zndnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:05:59.248333 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.248261 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f8466e73-ba2c-474b-892b-a2ca11bb9fad" (UID: "f8466e73-ba2c-474b-892b-a2ca11bb9fad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:05:59.347140 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.347114 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndnv\" (UniqueName: \"kubernetes.io/projected/f8466e73-ba2c-474b-892b-a2ca11bb9fad-kube-api-access-zndnv\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:59.347140 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.347140 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8466e73-ba2c-474b-892b-a2ca11bb9fad-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:59.347309 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.347149 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:05:59.347309 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:05:59.347161 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8466e73-ba2c-474b-892b-a2ca11bb9fad-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:00.089277 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:00.089187 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" Sep 10 15:06:00.089277 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:00.089222 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr" event={"ID":"f8466e73-ba2c-474b-892b-a2ca11bb9fad","Type":"ContainerDied","Data":"948282ccbb8d017ece9d4f2ff424d9ddc43e5aacfc8dba57bcf75f9910e53522"} Sep 10 15:06:00.089793 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:00.089283 2560 scope.go:117] "RemoveContainer" containerID="d6090e1121ba90e8d840bb07cca8a720ec1aa7ec7edcea2312375c8342080729" Sep 10 15:06:00.107538 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:00.107512 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr"] Sep 10 15:06:00.111394 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:00.111371 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e577f0-5d7c5b88b4-xjcpr"] Sep 10 15:06:01.674262 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:01.674225 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8466e73-ba2c-474b-892b-a2ca11bb9fad" path="/var/lib/kubelet/pods/f8466e73-ba2c-474b-892b-a2ca11bb9fad/volumes" Sep 10 15:06:05.361357 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:05.361315 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb"] Sep 10 15:06:05.361844 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:05.361519 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" containerID="cri-o://b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d" gracePeriod=30 Sep 10 15:06:07.090823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:07.090762 2560 patch_prober.go:28] interesting pod/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 15:06:07.091231 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:07.090849 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 15:06:07.132220 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:07.132188 2560 generic.go:356] "Generic (PLEG): container finished" podID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerID="8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13" exitCode=0 Sep 10 15:06:07.132375 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:07.132261 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" event={"ID":"39d35a27-7fee-4255-b582-ded92b48cd2f","Type":"ContainerDied","Data":"8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13"} Sep 10 15:06:07.132655 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:07.132642 2560 scope.go:117] "RemoveContainer" containerID="8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13" Sep 10 15:06:07.133678 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:07.133649 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:07.133741 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:07.133718 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:07.133776 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:07.133752 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:08.139184 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:08.139152 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" event={"ID":"39d35a27-7fee-4255-b582-ded92b48cd2f","Type":"ContainerStarted","Data":"9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113"} Sep 10 15:06:08.852779 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:08.852745 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:06:08.853086 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:08.852954 2560 patch_prober.go:28] interesting pod/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:06:08.853271 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:08.853246 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 15:06:09.853605 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:09.853120 2560 patch_prober.go:28] interesting pod/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:06:09.853605 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:09.853198 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 15:06:10.631431 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.631372 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:06:10.686995 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.686958 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-credential-socket\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687175 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687026 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bk2f\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-kube-api-access-6bk2f\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687175 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687062 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4a061b-907b-4597-a054-c8d89dffe728-istio-podinfo\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687175 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687108 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-certs\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687175 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687147 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4a061b-907b-4597-a054-c8d89dffe728-istiod-ca-cert\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687184 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-envoy\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687229 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:10.687424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687249 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-data\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687284 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-socket\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687308 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-istio-token\") pod \"be4a061b-907b-4597-a054-c8d89dffe728\" (UID: \"be4a061b-907b-4597-a054-c8d89dffe728\") " Sep 10 15:06:10.687649 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687419 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:10.687649 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687546 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4a061b-907b-4597-a054-c8d89dffe728-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:10.687649 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687569 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-data" (OuterVolumeSpecName: "istio-data") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:10.687745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687680 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.687745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687692 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4a061b-907b-4597-a054-c8d89dffe728-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.687745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687700 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.687745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687709 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.687745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.687714 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:10.689450 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.689419 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be4a061b-907b-4597-a054-c8d89dffe728-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:06:10.689721 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.689701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-kube-api-access-6bk2f" (OuterVolumeSpecName: "kube-api-access-6bk2f") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "kube-api-access-6bk2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:10.689973 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.689953 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:10.690032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.689991 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-istio-token" (OuterVolumeSpecName: "istio-token") pod "be4a061b-907b-4597-a054-c8d89dffe728" (UID: "be4a061b-907b-4597-a054-c8d89dffe728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:10.789111 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.789069 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bk2f\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-kube-api-access-6bk2f\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.789111 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.789114 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4a061b-907b-4597-a054-c8d89dffe728-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.789353 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.789130 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.789353 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.789144 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4a061b-907b-4597-a054-c8d89dffe728-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.789353 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.789157 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4a061b-907b-4597-a054-c8d89dffe728-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:10.853185 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.852680 2560 patch_prober.go:28] interesting pod/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:06:10.853185 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:10.852764 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 15:06:11.159982 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.159928 2560 generic.go:356] "Generic (PLEG): container finished" podID="be4a061b-907b-4597-a054-c8d89dffe728" containerID="b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d" exitCode=0 Sep 10 15:06:11.162509 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.160666 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" Sep 10 15:06:11.162509 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.160745 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" event={"ID":"be4a061b-907b-4597-a054-c8d89dffe728","Type":"ContainerDied","Data":"b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d"} Sep 10 15:06:11.162509 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.160970 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb" event={"ID":"be4a061b-907b-4597-a054-c8d89dffe728","Type":"ContainerDied","Data":"db0b0e9955de4c3bdad7def35e9c030dc0ac61cee25960944e6dc764d1824988"} Sep 10 15:06:11.162509 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.161015 2560 scope.go:117] "RemoveContainer" containerID="b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d" Sep 10 15:06:11.178486 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.178217 2560 scope.go:117] "RemoveContainer" containerID="cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23" Sep 10 15:06:11.188330 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.188312 2560 scope.go:117] "RemoveContainer" containerID="b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d" Sep 10 15:06:11.188614 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:11.188585 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d\": container with ID starting with b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d not found: ID does not exist" containerID="b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d" Sep 10 15:06:11.188706 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.188616 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d"} err="failed to get container status \"b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d\": rpc error: code = NotFound desc = could not find container \"b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d\": container with ID starting with b34faaf66a3773d7c4226af9babbda02e5680adca57a06e7d8861f2287a8ab0d not found: ID does not exist" Sep 10 15:06:11.188706 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.188638 2560 scope.go:117] "RemoveContainer" containerID="cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23" Sep 10 15:06:11.188956 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:11.188938 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23\": container with ID starting with cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23 not found: ID does not exist" containerID="cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23" Sep 10 15:06:11.189020 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.188961 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23"} err="failed to get container status \"cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23\": rpc error: code = NotFound desc = could not find container \"cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23\": container with ID starting with cb0ae93bb927aa0888bac68972e0caa40b41a9bd850d08948cba7c3c1087bb23 not found: ID does not exist" Sep 10 15:06:11.195490 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.195466 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb"] Sep 10 15:06:11.203843 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.203822 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e577f0-openshift-default-5fd4f68854-wzchb"] Sep 10 15:06:11.673861 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.673821 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be4a061b-907b-4597-a054-c8d89dffe728" path="/var/lib/kubelet/pods/be4a061b-907b-4597-a054-c8d89dffe728/volumes" Sep 10 15:06:11.852788 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.852754 2560 patch_prober.go:28] interesting pod/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:06:11.853107 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:11.853078 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 15:06:12.852959 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:12.852925 2560 patch_prober.go:28] interesting pod/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:06:12.853527 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:12.853495 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 15:06:13.857353 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:13.857323 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:06:13.857776 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:13.857637 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:06:13.858223 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:13.858204 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:06:23.484568 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:23.484539 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp"] Sep 10 15:06:23.485042 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:23.484776 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" containerID="cri-o://634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c" gracePeriod=30 Sep 10 15:06:24.364019 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.363981 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t"] Sep 10 15:06:24.364387 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364373 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" Sep 10 15:06:24.364454 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364389 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" Sep 10 15:06:24.364454 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364408 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be72303-e148-468e-949e-f9f58f8dc6c2" containerName="istio-proxy" Sep 10 15:06:24.364454 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364415 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be72303-e148-468e-949e-f9f58f8dc6c2" containerName="istio-proxy" Sep 10 15:06:24.364454 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364422 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8466e73-ba2c-474b-892b-a2ca11bb9fad" containerName="maas-api" Sep 10 15:06:24.364454 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364427 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8466e73-ba2c-474b-892b-a2ca11bb9fad" containerName="maas-api" Sep 10 15:06:24.364454 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364438 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" Sep 10 15:06:24.364454 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364447 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" Sep 10 15:06:24.364730 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364535 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8466e73-ba2c-474b-892b-a2ca11bb9fad" containerName="maas-api" Sep 10 15:06:24.364730 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364543 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" Sep 10 15:06:24.364730 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364548 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be72303-e148-468e-949e-f9f58f8dc6c2" containerName="istio-proxy" Sep 10 15:06:24.364730 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.364554 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be4a061b-907b-4597-a054-c8d89dffe728" containerName="istio-proxy" Sep 10 15:06:24.369347 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.369331 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.372076 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.372054 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-decc52-openshift-default-dockercfg-zzgnh\"" Sep 10 15:06:24.385653 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.385628 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t"] Sep 10 15:06:24.519705 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.519670 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-token\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.520237 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.519727 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-certs\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.520237 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.519825 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-data\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.520237 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.519872 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istiod-ca-cert\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.520237 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.519917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-socket\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.520237 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.519944 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-credential-socket\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.520237 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.520028 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-envoy\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.520237 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.520139 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-podinfo\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.520237 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.520165 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47ln\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-kube-api-access-q47ln\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.621580 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621551 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-token\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.621736 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621602 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-certs\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.621736 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-data\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.621736 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621638 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istiod-ca-cert\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.621736 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621670 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-socket\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.621736 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-credential-socket\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.621736 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621726 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-envoy\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.622088 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-podinfo\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.622088 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.621844 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47ln\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-kube-api-access-q47ln\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.622088 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.622034 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-certs\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.622272 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.622113 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-data\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.622329 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.622271 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-socket\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.622329 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.622287 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-credential-socket\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.622329 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.622299 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istiod-ca-cert\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.624099 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.624080 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-envoy\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.624315 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.624298 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-podinfo\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.630937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.630917 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-token\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.631138 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.631120 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47ln\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-kube-api-access-q47ln\") pod \"e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.680624 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.680602 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:24.954908 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.954777 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t"] Sep 10 15:06:24.957195 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:06:24.957165 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a935cb7_ecc4_42f3_9937_7b5ccf35e929.slice/crio-55fbda06a08067d70b91e625f8f3b782d959115e8f99d9972b0b6dbece5722e1 WatchSource:0}: Error finding container 55fbda06a08067d70b91e625f8f3b782d959115e8f99d9972b0b6dbece5722e1: Status 404 returned error can't find the container with id 55fbda06a08067d70b91e625f8f3b782d959115e8f99d9972b0b6dbece5722e1 Sep 10 15:06:24.959446 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.959413 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:24.959547 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.959478 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:24.959547 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:24.959526 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:25.234979 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:25.234894 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" event={"ID":"3a935cb7-ecc4-42f3-9937-7b5ccf35e929","Type":"ContainerStarted","Data":"ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea"} Sep 10 15:06:25.234979 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:25.234947 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" event={"ID":"3a935cb7-ecc4-42f3-9937-7b5ccf35e929","Type":"ContainerStarted","Data":"55fbda06a08067d70b91e625f8f3b782d959115e8f99d9972b0b6dbece5722e1"} Sep 10 15:06:25.259286 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:25.259238 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" podStartSLOduration=1.259222293 podStartE2EDuration="1.259222293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:25.257922648 +0000 UTC m=+1124.072505400" watchObservedRunningTime="2026-09-10 15:06:25.259222293 +0000 UTC m=+1124.073805041" Sep 10 15:06:25.681437 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:25.681406 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:26.686297 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:26.686266 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:27.245100 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:27.245066 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:27.245997 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:27.245978 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:28.731185 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.731163 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:06:28.756992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.756925 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-token\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.756992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.756970 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-certs\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.757184 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.757028 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-credential-socket\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.757184 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.757061 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-podinfo\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.757184 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.757082 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-data\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.757184 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.757113 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-envoy\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.757184 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.757142 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-socket\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.757184 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.757177 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zjt6\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-kube-api-access-2zjt6\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.757501 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.757204 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc0e1a4-20b3-4144-b4e1-382919c300da-istiod-ca-cert\") pod \"0dc0e1a4-20b3-4144-b4e1-382919c300da\" (UID: \"0dc0e1a4-20b3-4144-b4e1-382919c300da\") " Sep 10 15:06:28.757765 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.757736 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:28.758380 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.758082 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-data" (OuterVolumeSpecName: "istio-data") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:28.758380 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.758338 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:28.758588 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.758394 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc0e1a4-20b3-4144-b4e1-382919c300da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:28.758851 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.758671 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:28.760199 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.760171 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:06:28.760284 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.760198 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-kube-api-access-2zjt6" (OuterVolumeSpecName: "kube-api-access-2zjt6") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "kube-api-access-2zjt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:28.760892 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.760869 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:28.761445 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.761425 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-token" (OuterVolumeSpecName: "istio-token") pod "0dc0e1a4-20b3-4144-b4e1-382919c300da" (UID: "0dc0e1a4-20b3-4144-b4e1-382919c300da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:28.859042 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859009 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:28.859042 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859041 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:28.859289 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859057 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:28.859289 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859069 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:28.859289 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859081 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:28.859289 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859094 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zjt6\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-kube-api-access-2zjt6\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:28.859289 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859107 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dc0e1a4-20b3-4144-b4e1-382919c300da-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:28.859289 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859119 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc0e1a4-20b3-4144-b4e1-382919c300da-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:28.859289 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:28.859131 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc0e1a4-20b3-4144-b4e1-382919c300da-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:29.260538 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.260505 2560 generic.go:356] "Generic (PLEG): container finished" podID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerID="634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c" exitCode=0 Sep 10 15:06:29.260738 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.260573 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" Sep 10 15:06:29.260738 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.260591 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" event={"ID":"0dc0e1a4-20b3-4144-b4e1-382919c300da","Type":"ContainerDied","Data":"634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c"} Sep 10 15:06:29.260738 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.260645 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp" event={"ID":"0dc0e1a4-20b3-4144-b4e1-382919c300da","Type":"ContainerDied","Data":"76534be8ba9bd8f062e7ab2af4bb87431ae01035bbfdff1031b9cd5bcfee4371"} Sep 10 15:06:29.260738 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.260675 2560 scope.go:117] "RemoveContainer" containerID="634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c" Sep 10 15:06:29.272979 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.272954 2560 scope.go:117] "RemoveContainer" containerID="78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec" Sep 10 15:06:29.286280 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.285764 2560 scope.go:117] "RemoveContainer" containerID="634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c" Sep 10 15:06:29.287182 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:29.286670 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c\": container with ID starting with 634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c not found: ID does not exist" containerID="634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c" Sep 10 15:06:29.287182 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.286715 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c"} err="failed to get container status \"634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c\": rpc error: code = NotFound desc = could not find container \"634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c\": container with ID starting with 634a0e2ca4b1da1f14e690f236e4bef5413f54b218c4857ec0af09e0c2b81d2c not found: ID does not exist" Sep 10 15:06:29.287182 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.286745 2560 scope.go:117] "RemoveContainer" containerID="78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec" Sep 10 15:06:29.287953 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:29.287860 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec\": container with ID starting with 78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec not found: ID does not exist" containerID="78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec" Sep 10 15:06:29.287953 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.287893 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec"} err="failed to get container status \"78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec\": rpc error: code = NotFound desc = could not find container \"78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec\": container with ID starting with 78695e6a9c7c93b584dc1243f68f9babe5674c7e3d5533bc152e3c95fc43feec not found: ID does not exist" Sep 10 15:06:29.291655 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.291633 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp"] Sep 10 15:06:29.298567 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.298548 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6356e4-openshift-default-56ddcf8466-hzbkp"] Sep 10 15:06:29.664547 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:29.664517 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" path="/var/lib/kubelet/pods/0dc0e1a4-20b3-4144-b4e1-382919c300da/volumes" Sep 10 15:06:31.156204 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.156176 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v"] Sep 10 15:06:31.156707 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.156387 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" podUID="4e4229cb-b62e-4f38-b010-386602882887" containerName="maas-api" containerID="cri-o://647a9a6f7dab6923e34ac57ee1c221a7a23ced229922e8e13153cdc409d050b2" gracePeriod=30 Sep 10 15:06:31.273699 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.273666 2560 generic.go:356] "Generic (PLEG): container finished" podID="4e4229cb-b62e-4f38-b010-386602882887" containerID="647a9a6f7dab6923e34ac57ee1c221a7a23ced229922e8e13153cdc409d050b2" exitCode=0 Sep 10 15:06:31.273875 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.273736 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" event={"ID":"4e4229cb-b62e-4f38-b010-386602882887","Type":"ContainerDied","Data":"647a9a6f7dab6923e34ac57ee1c221a7a23ced229922e8e13153cdc409d050b2"} Sep 10 15:06:31.406856 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.406783 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:06:31.483279 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.483250 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-maas-api-tls\") pod \"4e4229cb-b62e-4f38-b010-386602882887\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " Sep 10 15:06:31.483452 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.483287 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl6lq\" (UniqueName: \"kubernetes.io/projected/4e4229cb-b62e-4f38-b010-386602882887-kube-api-access-pl6lq\") pod \"4e4229cb-b62e-4f38-b010-386602882887\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " Sep 10 15:06:31.483452 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.483341 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-metrics-certs\") pod \"4e4229cb-b62e-4f38-b010-386602882887\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " Sep 10 15:06:31.483452 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.483403 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4229cb-b62e-4f38-b010-386602882887-openshift-service-ca\") pod \"4e4229cb-b62e-4f38-b010-386602882887\" (UID: \"4e4229cb-b62e-4f38-b010-386602882887\") " Sep 10 15:06:31.483850 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.483824 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4229cb-b62e-4f38-b010-386602882887-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e4229cb-b62e-4f38-b010-386602882887" (UID: "4e4229cb-b62e-4f38-b010-386602882887"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:31.485483 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.485458 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4229cb-b62e-4f38-b010-386602882887-kube-api-access-pl6lq" (OuterVolumeSpecName: "kube-api-access-pl6lq") pod "4e4229cb-b62e-4f38-b010-386602882887" (UID: "4e4229cb-b62e-4f38-b010-386602882887"). InnerVolumeSpecName "kube-api-access-pl6lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:31.485568 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.485460 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e4229cb-b62e-4f38-b010-386602882887" (UID: "4e4229cb-b62e-4f38-b010-386602882887"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:06:31.485568 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.485483 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e4229cb-b62e-4f38-b010-386602882887" (UID: "4e4229cb-b62e-4f38-b010-386602882887"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:06:31.584251 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.584214 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4229cb-b62e-4f38-b010-386602882887-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:31.584251 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.584248 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:31.584251 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.584258 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl6lq\" (UniqueName: \"kubernetes.io/projected/4e4229cb-b62e-4f38-b010-386602882887-kube-api-access-pl6lq\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:31.584482 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:31.584267 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4229cb-b62e-4f38-b010-386602882887-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:32.280322 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:32.280282 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" event={"ID":"4e4229cb-b62e-4f38-b010-386602882887","Type":"ContainerDied","Data":"38c1f55c4204d675ace83a815d74f3764144288135140f6db6c228406cb1ce48"} Sep 10 15:06:32.280722 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:32.280373 2560 scope.go:117] "RemoveContainer" containerID="647a9a6f7dab6923e34ac57ee1c221a7a23ced229922e8e13153cdc409d050b2" Sep 10 15:06:32.280722 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:32.280303 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v" Sep 10 15:06:32.299471 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:32.299444 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v"] Sep 10 15:06:32.305303 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:32.305281 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-411a8086-b6c49f4d7-2tz5v"] Sep 10 15:06:33.668163 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:33.668127 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e4229cb-b62e-4f38-b010-386602882887" path="/var/lib/kubelet/pods/4e4229cb-b62e-4f38-b010-386602882887/volumes" Sep 10 15:06:38.587874 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:38.587841 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9"] Sep 10 15:06:38.588332 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:38.588150 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" containerID="cri-o://9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113" gracePeriod=30 Sep 10 15:06:42.246471 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:42.246438 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:06:42.246866 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:42.246500 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:06:42.331145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:42.331112 2560 generic.go:356] "Generic (PLEG): container finished" podID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerID="ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea" exitCode=0 Sep 10 15:06:42.331310 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:42.331183 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" event={"ID":"3a935cb7-ecc4-42f3-9937-7b5ccf35e929","Type":"ContainerDied","Data":"ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea"} Sep 10 15:06:42.331568 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:42.331552 2560 scope.go:117] "RemoveContainer" containerID="ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea" Sep 10 15:06:42.332574 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:42.332536 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:42.332667 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:42.332613 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:42.332667 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:42.332652 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:43.338201 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.338163 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" event={"ID":"3a935cb7-ecc4-42f3-9937-7b5ccf35e929","Type":"ContainerStarted","Data":"71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0"} Sep 10 15:06:43.681248 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.681215 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:43.681439 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.681417 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:06:43.681514 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.681472 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:06:43.851081 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.851023 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:06:43.902437 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902412 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv4k9\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-kube-api-access-kv4k9\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902575 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902469 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-envoy\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902575 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902512 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-token\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902575 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902540 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-certs\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902714 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902576 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-data\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902714 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902617 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-socket\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902714 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902671 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-podinfo\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902894 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902721 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-credential-socket\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902894 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902751 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d35a27-7fee-4255-b582-ded92b48cd2f-istiod-ca-cert\") pod \"39d35a27-7fee-4255-b582-ded92b48cd2f\" (UID: \"39d35a27-7fee-4255-b582-ded92b48cd2f\") " Sep 10 15:06:43.902894 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902851 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:43.903049 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.902896 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:43.903049 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.903023 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:43.903147 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.903089 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:43.903212 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.903157 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:43.903212 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.903177 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:43.903212 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.903190 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:43.903373 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.903212 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:43.903373 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.903286 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d35a27-7fee-4255-b582-ded92b48cd2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:06:43.905150 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.905109 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-kube-api-access-kv4k9" (OuterVolumeSpecName: "kube-api-access-kv4k9") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "kube-api-access-kv4k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:43.905279 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.905220 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:06:43.905461 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.905434 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:06:43.905461 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:43.905436 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39d35a27-7fee-4255-b582-ded92b48cd2f" (UID: "39d35a27-7fee-4255-b582-ded92b48cd2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:06:44.003960 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.003931 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:44.003960 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.003956 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:44.003960 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.003965 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d35a27-7fee-4255-b582-ded92b48cd2f-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:44.004181 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.003973 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d35a27-7fee-4255-b582-ded92b48cd2f-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:44.004181 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.003985 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4k9\" (UniqueName: \"kubernetes.io/projected/39d35a27-7fee-4255-b582-ded92b48cd2f-kube-api-access-kv4k9\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:06:44.345268 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.345232 2560 generic.go:356] "Generic (PLEG): container finished" podID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerID="9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113" exitCode=0 Sep 10 15:06:44.345715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.345275 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" Sep 10 15:06:44.345715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.345324 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" event={"ID":"39d35a27-7fee-4255-b582-ded92b48cd2f","Type":"ContainerDied","Data":"9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113"} Sep 10 15:06:44.345715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.345375 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9" event={"ID":"39d35a27-7fee-4255-b582-ded92b48cd2f","Type":"ContainerDied","Data":"9e289a535eb51fe3ff75f8c871c486d414e9440eabb392d16ffdf64482a28c47"} Sep 10 15:06:44.345715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.345397 2560 scope.go:117] "RemoveContainer" containerID="9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113" Sep 10 15:06:44.356260 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.356237 2560 scope.go:117] "RemoveContainer" containerID="8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13" Sep 10 15:06:44.370334 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.370291 2560 scope.go:117] "RemoveContainer" containerID="9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113" Sep 10 15:06:44.370665 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:44.370636 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113\": container with ID starting with 9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113 not found: ID does not exist" containerID="9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113" Sep 10 15:06:44.370771 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.370676 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113"} err="failed to get container status \"9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113\": rpc error: code = NotFound desc = could not find container \"9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113\": container with ID starting with 9915eed878f5ab264c9e9407c547a811db5be392bd45dc4e98865a5904292113 not found: ID does not exist" Sep 10 15:06:44.370771 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.370701 2560 scope.go:117] "RemoveContainer" containerID="8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13" Sep 10 15:06:44.371056 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:44.371028 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13\": container with ID starting with 8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13 not found: ID does not exist" containerID="8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13" Sep 10 15:06:44.371131 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.371064 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13"} err="failed to get container status \"8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13\": rpc error: code = NotFound desc = could not find container \"8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13\": container with ID starting with 8b5811e64b575f2db2279e055367063f44c3bc4bcbdfb3db002417fd3f26df13 not found: ID does not exist" Sep 10 15:06:44.382558 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.382523 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9"] Sep 10 15:06:44.388276 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.388253 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-411a8086-openshift-default-65dc6b9568-vfgx9"] Sep 10 15:06:44.681813 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.681761 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:06:44.681990 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:44.681833 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:06:45.668711 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:45.668674 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" path="/var/lib/kubelet/pods/39d35a27-7fee-4255-b582-ded92b48cd2f/volumes" Sep 10 15:06:45.681196 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:45.681162 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:06:45.681310 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:45.681220 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:06:46.681868 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:46.681831 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:06:46.682264 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:46.681884 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:06:47.681160 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:47.681126 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:06:47.681364 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:47.681184 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:06:48.685570 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:48.685539 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:48.686015 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:48.685818 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:48.686498 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:48.686479 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:06:52.695207 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695167 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn"] Sep 10 15:06:52.695578 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695544 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4229cb-b62e-4f38-b010-386602882887" containerName="maas-api" Sep 10 15:06:52.695578 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695556 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4229cb-b62e-4f38-b010-386602882887" containerName="maas-api" Sep 10 15:06:52.695578 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695566 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" Sep 10 15:06:52.695578 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695571 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" Sep 10 15:06:52.695717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695604 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" Sep 10 15:06:52.695717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695610 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" Sep 10 15:06:52.695717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695626 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" Sep 10 15:06:52.695717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695632 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" Sep 10 15:06:52.695717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695683 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" Sep 10 15:06:52.695717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695690 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" Sep 10 15:06:52.695717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695696 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" Sep 10 15:06:52.695717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.695705 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4229cb-b62e-4f38-b010-386602882887" containerName="maas-api" Sep 10 15:06:52.700188 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.700166 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.702827 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.702806 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d5a2df-openshift-default-dockercfg-2bc2j\"" Sep 10 15:06:52.717018 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.716995 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn"] Sep 10 15:06:52.784739 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.784703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-credential-socket\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.784934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.784755 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfq8f\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-kube-api-access-wfq8f\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.784934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.784911 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-data\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.785060 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.784950 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-envoy\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.785119 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.785052 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efce121-55ac-4c92-8717-88557ae30e1a-istio-podinfo\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.785119 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.785109 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-istio-token\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.785223 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.785151 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-socket\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.785223 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.785200 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-certs\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.785322 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.785243 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efce121-55ac-4c92-8717-88557ae30e1a-istiod-ca-cert\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886249 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886211 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-istio-token\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886262 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-socket\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886500 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886448 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-certs\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886563 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efce121-55ac-4c92-8717-88557ae30e1a-istiod-ca-cert\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886623 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886564 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-credential-socket\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886623 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886609 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfq8f\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-kube-api-access-wfq8f\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886722 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886653 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-socket\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886782 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886727 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-data\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886782 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886743 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-certs\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886782 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886766 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-envoy\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.886966 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886843 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efce121-55ac-4c92-8717-88557ae30e1a-istio-podinfo\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.887030 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.886992 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-credential-socket\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.887071 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.887044 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-data\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.887283 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.887263 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efce121-55ac-4c92-8717-88557ae30e1a-istiod-ca-cert\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.888992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.888969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-envoy\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.889183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.889168 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efce121-55ac-4c92-8717-88557ae30e1a-istio-podinfo\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.899605 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.899580 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-istio-token\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.900053 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.900035 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfq8f\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-kube-api-access-wfq8f\") pod \"e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:52.936455 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.936421 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd"] Sep 10 15:06:52.936965 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.936948 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" Sep 10 15:06:52.936965 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.936965 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d35a27-7fee-4255-b582-ded92b48cd2f" containerName="istio-proxy" Sep 10 15:06:52.937105 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.937026 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc0e1a4-20b3-4144-b4e1-382919c300da" containerName="istio-proxy" Sep 10 15:06:52.941000 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.940981 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.943847 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.943827 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-0a9de4-openshift-default-dockercfg-2l5cl\"" Sep 10 15:06:52.953754 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.953733 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd"] Sep 10 15:06:52.987897 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.987866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-certs\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.988056 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.987903 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-envoy\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.988056 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.987928 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-data\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.988056 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.987943 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-token\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.988160 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.988044 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-socket\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.988160 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.988089 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad40d8-cdfb-4d49-9604-0fdab9515172-istiod-ca-cert\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.988160 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.988115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lggg\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-kube-api-access-4lggg\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.988261 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.988160 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-credential-socket\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:52.988295 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:52.988278 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-podinfo\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.010957 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.010926 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.088942 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-podinfo\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089010 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-certs\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089039 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-envoy\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089068 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-data\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089092 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-token\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089143 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-socket\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089171 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad40d8-cdfb-4d49-9604-0fdab9515172-istiod-ca-cert\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089196 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lggg\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-kube-api-access-4lggg\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-credential-socket\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.089929 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.089596 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-data\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.090041 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.090017 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-credential-socket\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.090162 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.090121 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad40d8-cdfb-4d49-9604-0fdab9515172-istiod-ca-cert\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.090605 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.090566 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-certs\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.090707 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.090640 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-socket\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.092747 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.092721 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-envoy\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.092844 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.092754 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-podinfo\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.099672 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.099624 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-token\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.102983 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.102937 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lggg\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-kube-api-access-4lggg\") pod \"e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.256496 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.256412 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:53.308092 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.308065 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn"] Sep 10 15:06:53.312092 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:06:53.311603 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efce121_55ac_4c92_8717_88557ae30e1a.slice/crio-4ee43301e6ffca8cdb756f87f3c8d16328e848b3875077e74f37815eee7ddffd WatchSource:0}: Error finding container 4ee43301e6ffca8cdb756f87f3c8d16328e848b3875077e74f37815eee7ddffd: Status 404 returned error can't find the container with id 4ee43301e6ffca8cdb756f87f3c8d16328e848b3875077e74f37815eee7ddffd Sep 10 15:06:53.314222 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.314189 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:53.314324 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.314253 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:53.314324 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.314280 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:53.399229 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.399200 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" event={"ID":"6efce121-55ac-4c92-8717-88557ae30e1a","Type":"ContainerStarted","Data":"4ee43301e6ffca8cdb756f87f3c8d16328e848b3875077e74f37815eee7ddffd"} Sep 10 15:06:53.542222 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.542196 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd"] Sep 10 15:06:53.544379 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:06:53.544354 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecad40d8_cdfb_4d49_9604_0fdab9515172.slice/crio-f22676c271bfc8c68d8f834f4206791f481a40b4ac498b8a35fc74df6f4729bb WatchSource:0}: Error finding container f22676c271bfc8c68d8f834f4206791f481a40b4ac498b8a35fc74df6f4729bb: Status 404 returned error can't find the container with id f22676c271bfc8c68d8f834f4206791f481a40b4ac498b8a35fc74df6f4729bb Sep 10 15:06:53.547206 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.547152 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:53.547298 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.547246 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:53.547298 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:53.547289 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:06:54.406159 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:54.406121 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" event={"ID":"ecad40d8-cdfb-4d49-9604-0fdab9515172","Type":"ContainerStarted","Data":"76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7"} Sep 10 15:06:54.406628 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:54.406180 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" event={"ID":"ecad40d8-cdfb-4d49-9604-0fdab9515172","Type":"ContainerStarted","Data":"f22676c271bfc8c68d8f834f4206791f481a40b4ac498b8a35fc74df6f4729bb"} Sep 10 15:06:54.407697 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:54.407676 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" event={"ID":"6efce121-55ac-4c92-8717-88557ae30e1a","Type":"ContainerStarted","Data":"b4abc51a2d514a4d1d5fcb4ad3b75b27b8a6ec6e109da6f087374652017010fa"} Sep 10 15:06:54.452367 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:54.452319 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" podStartSLOduration=2.45230419 podStartE2EDuration="2.45230419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:54.426755629 +0000 UTC m=+1153.241338392" watchObservedRunningTime="2026-09-10 15:06:54.45230419 +0000 UTC m=+1153.266886940" Sep 10 15:06:54.452899 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:54.452873 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" podStartSLOduration=2.452865369 podStartE2EDuration="2.452865369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:54.450640204 +0000 UTC m=+1153.265223005" watchObservedRunningTime="2026-09-10 15:06:54.452865369 +0000 UTC m=+1153.267448121" Sep 10 15:06:55.011383 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:55.011348 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:55.016405 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:55.016381 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:55.257341 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:55.257307 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:55.262272 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:55.262201 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:55.412450 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:55.412413 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:55.412860 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:55.412477 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:55.413228 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:55.413209 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:06:55.413305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:55.413259 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:06:57.651234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.651189 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw"] Sep 10 15:06:57.655626 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.655608 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.658726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.658704 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d5a2df-serving-cert\"" Sep 10 15:06:57.676047 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.675894 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw"] Sep 10 15:06:57.736103 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.736073 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.736242 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.736110 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9mdk\" (UniqueName: \"kubernetes.io/projected/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-kube-api-access-r9mdk\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.736242 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.736216 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.736322 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.736277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.837651 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.837617 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.837831 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.837657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9mdk\" (UniqueName: \"kubernetes.io/projected/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-kube-api-access-r9mdk\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.837831 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.837711 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.837831 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.837744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.837831 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:57.837772 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-serving-cert: secret "maas-api-e2e-shared-b-w3-d5a2df-serving-cert" not found Sep 10 15:06:57.838022 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:57.837892 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls podName:3f241998-59ca-4fb4-91fb-59a2dc7c37c5 nodeName:}" failed. No retries permitted until 2026-09-10 15:06:58.337869489 +0000 UTC m=+1157.152452221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" (UID: "3f241998-59ca-4fb4-91fb-59a2dc7c37c5") : secret "maas-api-e2e-shared-b-w3-d5a2df-serving-cert" not found Sep 10 15:06:57.838403 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.838371 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.840283 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.840261 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:57.852770 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:57.852744 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9mdk\" (UniqueName: \"kubernetes.io/projected/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-kube-api-access-r9mdk\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:58.343075 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.343043 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:58.345480 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.345456 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:58.572336 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.572297 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:58.687122 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.687093 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf"] Sep 10 15:06:58.691852 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.691832 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.696085 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.696061 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-0a9de4-serving-cert\"" Sep 10 15:06:58.711716 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.711685 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf"] Sep 10 15:06:58.848050 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.848022 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.848214 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.848078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.848214 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.848140 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssk7c\" (UniqueName: \"kubernetes.io/projected/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-kube-api-access-ssk7c\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.848214 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.848186 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.852712 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.852689 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw"] Sep 10 15:06:58.855186 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:06:58.855151 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f241998_59ca_4fb4_91fb_59a2dc7c37c5.slice/crio-1a1f2270b6b3b71dab84697dd33d60f29545a01dd74c3205cb9446bc8d03b4bd WatchSource:0}: Error finding container 1a1f2270b6b3b71dab84697dd33d60f29545a01dd74c3205cb9446bc8d03b4bd: Status 404 returned error can't find the container with id 1a1f2270b6b3b71dab84697dd33d60f29545a01dd74c3205cb9446bc8d03b4bd Sep 10 15:06:58.949476 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.949414 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.949476 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.949460 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.949654 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.949506 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssk7c\" (UniqueName: \"kubernetes.io/projected/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-kube-api-access-ssk7c\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.949654 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.949524 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.949654 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:58.949622 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-serving-cert: secret "maas-api-e2e-shared-a-w2-0a9de4-serving-cert" not found Sep 10 15:06:58.949845 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:58.949680 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls podName:e3449fdb-c3de-4a7d-83e0-f0dd60a17903 nodeName:}" failed. No retries permitted until 2026-09-10 15:06:59.449661933 +0000 UTC m=+1158.264244662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls") pod "maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" (UID: "e3449fdb-c3de-4a7d-83e0-f0dd60a17903") : secret "maas-api-e2e-shared-a-w2-0a9de4-serving-cert" not found Sep 10 15:06:58.950191 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.950172 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.951868 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.951839 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:58.964216 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:58.964192 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssk7c\" (UniqueName: \"kubernetes.io/projected/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-kube-api-access-ssk7c\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:59.433855 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:59.433789 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" event={"ID":"3f241998-59ca-4fb4-91fb-59a2dc7c37c5","Type":"ContainerStarted","Data":"dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e"} Sep 10 15:06:59.434041 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:59.433866 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" event={"ID":"3f241998-59ca-4fb4-91fb-59a2dc7c37c5","Type":"ContainerStarted","Data":"1a1f2270b6b3b71dab84697dd33d60f29545a01dd74c3205cb9446bc8d03b4bd"} Sep 10 15:06:59.434041 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:59.433936 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:06:59.454989 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:59.454962 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:06:59.455135 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:59.455088 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-serving-cert: secret "maas-api-e2e-shared-a-w2-0a9de4-serving-cert" not found Sep 10 15:06:59.455182 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:06:59.455144 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls podName:e3449fdb-c3de-4a7d-83e0-f0dd60a17903 nodeName:}" failed. No retries permitted until 2026-09-10 15:07:00.455130304 +0000 UTC m=+1159.269713032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls") pod "maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" (UID: "e3449fdb-c3de-4a7d-83e0-f0dd60a17903") : secret "maas-api-e2e-shared-a-w2-0a9de4-serving-cert" not found Sep 10 15:06:59.458654 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:06:59.458603 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" podStartSLOduration=2.050707302 podStartE2EDuration="2.458584255s" totalImagesPullingTime="407.876953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:59.456855438 +0000 UTC m=+1158.271438189" watchObservedRunningTime="2026-09-10 15:06:59.458584255 +0000 UTC m=+1158.273167009" Sep 10 15:07:00.466067 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:00.466021 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:07:00.468347 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:00.468327 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:07:00.504460 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:00.504431 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:07:00.777029 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:00.777006 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf"] Sep 10 15:07:00.778768 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:07:00.778741 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3449fdb_c3de_4a7d_83e0_f0dd60a17903.slice/crio-ec8b91a86aed066eda46d7f6f5fbf9b1ff829712f80f0acbff28b61f757ec0c8 WatchSource:0}: Error finding container ec8b91a86aed066eda46d7f6f5fbf9b1ff829712f80f0acbff28b61f757ec0c8: Status 404 returned error can't find the container with id ec8b91a86aed066eda46d7f6f5fbf9b1ff829712f80f0acbff28b61f757ec0c8 Sep 10 15:07:01.446715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:01.446684 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" event={"ID":"e3449fdb-c3de-4a7d-83e0-f0dd60a17903","Type":"ContainerStarted","Data":"8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d"} Sep 10 15:07:01.446919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:01.446732 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" event={"ID":"e3449fdb-c3de-4a7d-83e0-f0dd60a17903","Type":"ContainerStarted","Data":"ec8b91a86aed066eda46d7f6f5fbf9b1ff829712f80f0acbff28b61f757ec0c8"} Sep 10 15:07:01.446919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:01.446764 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:07:01.470844 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:01.470778 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" podStartSLOduration=2.93495723 podStartE2EDuration="3.470762642s" totalImagesPullingTime="535.805412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:01.468999139 +0000 UTC m=+1160.283581905" watchObservedRunningTime="2026-09-10 15:07:01.470762642 +0000 UTC m=+1160.285345393" Sep 10 15:07:05.446331 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:05.446262 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:07:07.469114 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:07.469082 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:07:14.533079 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.532998 2560 generic.go:356] "Generic (PLEG): container finished" podID="6efce121-55ac-4c92-8717-88557ae30e1a" containerID="b4abc51a2d514a4d1d5fcb4ad3b75b27b8a6ec6e109da6f087374652017010fa" exitCode=0 Sep 10 15:07:14.533539 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.533075 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" event={"ID":"6efce121-55ac-4c92-8717-88557ae30e1a","Type":"ContainerDied","Data":"b4abc51a2d514a4d1d5fcb4ad3b75b27b8a6ec6e109da6f087374652017010fa"} Sep 10 15:07:14.533539 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.533494 2560 scope.go:117] "RemoveContainer" containerID="b4abc51a2d514a4d1d5fcb4ad3b75b27b8a6ec6e109da6f087374652017010fa" Sep 10 15:07:14.534737 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.534701 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:14.534836 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.534788 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:14.534889 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.534880 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:14.534959 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.534941 2560 generic.go:356] "Generic (PLEG): container finished" podID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerID="76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7" exitCode=0 Sep 10 15:07:14.535001 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.534973 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" event={"ID":"ecad40d8-cdfb-4d49-9604-0fdab9515172","Type":"ContainerDied","Data":"76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7"} Sep 10 15:07:14.535344 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.535331 2560 scope.go:117] "RemoveContainer" containerID="76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7" Sep 10 15:07:14.536284 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.536260 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:14.536336 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.536317 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:14.536371 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:14.536351 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:15.541856 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:15.541814 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" event={"ID":"ecad40d8-cdfb-4d49-9604-0fdab9515172","Type":"ContainerStarted","Data":"52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c"} Sep 10 15:07:15.543672 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:15.543649 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" event={"ID":"6efce121-55ac-4c92-8717-88557ae30e1a","Type":"ContainerStarted","Data":"80612fe929c7afcd2dc4e1ad8a40d4979a71b38d97248c96d333d5d8a971d270"} Sep 10 15:07:16.011790 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:16.011748 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:07:16.012078 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:16.012053 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:07:16.012174 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:16.012124 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 15:07:16.256985 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:16.256952 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:07:16.256985 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:16.256986 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:07:16.257250 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:16.257033 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:07:17.011688 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:17.011653 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:07:17.012096 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:17.011725 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 15:07:17.257780 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:17.257736 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:07:17.257969 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:17.257821 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:07:18.012305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:18.012266 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:07:18.012704 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:18.012321 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 15:07:18.257209 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:18.257163 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:07:18.257428 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:18.257235 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:07:19.011462 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:19.011423 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:07:19.011639 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:19.011488 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 15:07:19.257293 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:19.257249 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:07:19.257768 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:19.257318 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:07:20.012974 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:20.012942 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:07:20.013151 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:20.013006 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 15:07:20.257873 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:20.257831 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 15:07:20.258356 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:20.257912 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 15:07:21.011523 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.011489 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:07:21.011706 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.011540 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 15:07:21.261505 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.261476 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:07:21.261907 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.261838 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:07:21.262582 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.262568 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:07:21.924477 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.924444 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk"] Sep 10 15:07:21.927576 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.927546 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:21.930109 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.930082 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-1f985c-openshift-default-dockercfg-ls2w6\"" Sep 10 15:07:21.953818 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:21.953767 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk"] Sep 10 15:07:22.008687 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.008643 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-token\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.008889 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.008717 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-podinfo\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.008889 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.008779 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-data\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.008889 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.008837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-credential-socket\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.009036 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.008892 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d602a4c-1d84-438b-89dc-09d049ea4499-istiod-ca-cert\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.009036 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.008935 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-socket\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.009036 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.008976 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-envoy\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.009036 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.009019 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-certs\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.009338 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.009256 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwz2\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-kube-api-access-pzwz2\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.048231 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.048196 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:07:22.048866 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.048844 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:07:22.049992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.049970 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:07:22.110160 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110126 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwz2\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-kube-api-access-pzwz2\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110344 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110175 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-token\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110344 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110211 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-podinfo\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110344 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-data\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110344 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110286 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-credential-socket\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110344 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110328 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d602a4c-1d84-438b-89dc-09d049ea4499-istiod-ca-cert\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110621 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110386 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-socket\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110621 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-envoy\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110621 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-certs\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110850 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110783 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-data\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.110850 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.110830 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-certs\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.111152 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.111047 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-socket\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.111152 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.111142 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-credential-socket\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.111295 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.111229 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d602a4c-1d84-438b-89dc-09d049ea4499-istiod-ca-cert\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.113487 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.113460 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-envoy\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.113592 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.113509 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-podinfo\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.122815 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.122780 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-token\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.123178 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.123160 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwz2\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-kube-api-access-pzwz2\") pod \"e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.239076 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.238990 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:22.515873 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.515845 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk"] Sep 10 15:07:22.517981 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:07:22.517955 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d602a4c_1d84_438b_89dc_09d049ea4499.slice/crio-e021a9f6e6a037f54b9641ef9772b55746f3d408c58a43d9729c362aa01ef8e5 WatchSource:0}: Error finding container e021a9f6e6a037f54b9641ef9772b55746f3d408c58a43d9729c362aa01ef8e5: Status 404 returned error can't find the container with id e021a9f6e6a037f54b9641ef9772b55746f3d408c58a43d9729c362aa01ef8e5 Sep 10 15:07:22.520271 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.520236 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:22.520365 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.520302 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:22.520365 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.520330 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:22.594005 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.593969 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" event={"ID":"8d602a4c-1d84-438b-89dc-09d049ea4499","Type":"ContainerStarted","Data":"dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b"} Sep 10 15:07:22.594141 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.594036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" event={"ID":"8d602a4c-1d84-438b-89dc-09d049ea4499","Type":"ContainerStarted","Data":"e021a9f6e6a037f54b9641ef9772b55746f3d408c58a43d9729c362aa01ef8e5"} Sep 10 15:07:22.622024 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:22.621977 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" podStartSLOduration=1.62195643 podStartE2EDuration="1.62195643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:22.619597829 +0000 UTC m=+1181.434180605" watchObservedRunningTime="2026-09-10 15:07:22.62195643 +0000 UTC m=+1181.436539180" Sep 10 15:07:23.239927 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:23.239880 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:23.244450 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:23.244424 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:23.599749 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:23.599661 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:23.600630 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:23.600613 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:39.640953 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:39.640912 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:07:39.641355 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:39.640970 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:07:40.703683 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:40.703648 2560 generic.go:356] "Generic (PLEG): container finished" podID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerID="dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b" exitCode=0 Sep 10 15:07:40.704231 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:40.703724 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" event={"ID":"8d602a4c-1d84-438b-89dc-09d049ea4499","Type":"ContainerDied","Data":"dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b"} Sep 10 15:07:40.704303 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:40.704262 2560 scope.go:117] "RemoveContainer" containerID="dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b" Sep 10 15:07:40.705393 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:40.705362 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:40.705489 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:40.705431 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:40.705489 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:40.705463 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:41.710919 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:41.710886 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" event={"ID":"8d602a4c-1d84-438b-89dc-09d049ea4499","Type":"ContainerStarted","Data":"1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907"} Sep 10 15:07:41.711623 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:41.711602 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:07:41.715507 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:41.715484 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:07:42.239523 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:42.239487 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:42.239701 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:42.239656 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 15:07:42.239750 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:42.239700 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 15:07:43.240009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:43.239972 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 15:07:43.240398 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:43.240055 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 15:07:44.239819 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:44.239766 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 15:07:44.240001 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:44.239860 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 15:07:45.240278 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:45.240243 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 15:07:45.240710 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:45.240318 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 15:07:46.240369 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:46.240332 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 15:07:46.240754 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:46.240392 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 15:07:47.264504 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:47.264462 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:47.264988 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:47.264761 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:47.265409 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:47.265389 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:07:58.511050 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.511018 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj"] Sep 10 15:07:58.516074 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.516054 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.519216 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.519192 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5c801c28-openshift-default-dockercfg-zlzrx\"" Sep 10 15:07:58.539338 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.539310 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj"] Sep 10 15:07:58.578710 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578679 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istiod-ca-cert\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.578852 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578719 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-podinfo\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.578852 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578767 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-credential-socket\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.578852 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578785 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-certs\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.579032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578859 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-socket\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.579032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578885 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-envoy\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.579032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578910 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv497\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-kube-api-access-gv497\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.579032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578935 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-token\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.579032 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.578988 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-data\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680263 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680171 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-credential-socket\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680429 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680288 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-certs\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680429 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680324 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-socket\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680429 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680341 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-envoy\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680429 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680381 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv497\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-kube-api-access-gv497\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680429 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680419 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-token\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680684 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680477 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-data\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680684 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680541 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-credential-socket\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680684 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680548 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istiod-ca-cert\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680684 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680647 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-podinfo\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.680950 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680729 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-socket\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.681014 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.680967 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-data\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.681212 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.681190 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-certs\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.681294 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.681237 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istiod-ca-cert\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.682671 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.682648 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-envoy\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.683097 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.683081 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-podinfo\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.689187 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.689168 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-token\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.689404 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.689383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv497\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-kube-api-access-gv497\") pod \"e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:58.826524 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:58.826449 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:59.107458 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:59.107427 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj"] Sep 10 15:07:59.109155 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:07:59.109130 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ce25c4_81a5_4360_a2f0_a4f534ada6ca.slice/crio-b3037e6d2af98cb9ad24668ef220da594febcb081ac8c61e2a4bdc2db7c38296 WatchSource:0}: Error finding container b3037e6d2af98cb9ad24668ef220da594febcb081ac8c61e2a4bdc2db7c38296: Status 404 returned error can't find the container with id b3037e6d2af98cb9ad24668ef220da594febcb081ac8c61e2a4bdc2db7c38296 Sep 10 15:07:59.111556 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:59.111521 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:59.111632 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:59.111585 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:59.111632 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:59.111613 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:07:59.811404 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:59.811369 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" event={"ID":"37ce25c4-81a5-4360-a2f0-a4f534ada6ca","Type":"ContainerStarted","Data":"1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277"} Sep 10 15:07:59.811767 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:59.811425 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" event={"ID":"37ce25c4-81a5-4360-a2f0-a4f534ada6ca","Type":"ContainerStarted","Data":"b3037e6d2af98cb9ad24668ef220da594febcb081ac8c61e2a4bdc2db7c38296"} Sep 10 15:07:59.827426 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:59.827407 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:07:59.833665 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:07:59.833614 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" podStartSLOduration=1.8336013370000002 podStartE2EDuration="1.833601337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:59.832417275 +0000 UTC m=+1218.647000026" watchObservedRunningTime="2026-09-10 15:07:59.833601337 +0000 UTC m=+1218.648184089" Sep 10 15:08:00.832505 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:00.832468 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:08:01.821717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:01.821686 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:08:01.822503 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:01.822481 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:08:25.948305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:25.948222 2560 generic.go:356] "Generic (PLEG): container finished" podID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerID="1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277" exitCode=0 Sep 10 15:08:25.948305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:25.948279 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" event={"ID":"37ce25c4-81a5-4360-a2f0-a4f534ada6ca","Type":"ContainerDied","Data":"1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277"} Sep 10 15:08:25.948811 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:25.948657 2560 scope.go:117] "RemoveContainer" containerID="1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277" Sep 10 15:08:25.952412 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:25.952373 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:08:25.952550 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:25.952450 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:08:25.952550 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:25.952490 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:08:26.958203 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:26.958165 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" event={"ID":"37ce25c4-81a5-4360-a2f0-a4f534ada6ca","Type":"ContainerStarted","Data":"b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf"} Sep 10 15:08:27.827562 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:27.827528 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:08:27.828679 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:27.828658 2560 patch_prober.go:28] interesting pod/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 10 15:08:27.828787 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:27.828712 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 10 15:08:28.827358 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:28.827322 2560 patch_prober.go:28] interesting pod/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 10 15:08:28.827873 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:28.827391 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 10 15:08:29.827544 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:29.827505 2560 patch_prober.go:28] interesting pod/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 10 15:08:29.828017 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:29.827561 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 10 15:08:30.827335 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:30.827299 2560 patch_prober.go:28] interesting pod/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 10 15:08:30.827519 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:30.827363 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 10 15:08:31.821618 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:31.821586 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:08:31.827310 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:31.827286 2560 patch_prober.go:28] interesting pod/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Sep 10 15:08:31.827388 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:31.827336 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Sep 10 15:08:32.831009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:32.830976 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:08:32.832073 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:08:32.832054 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:10:24.103320 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:24.103282 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj"] Sep 10 15:10:24.103856 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:24.103498 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" containerID="cri-o://b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf" gracePeriod=30 Sep 10 15:10:29.355016 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.354991 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:10:29.497902 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.497826 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-credential-socket\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.497902 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.497865 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-token\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.497902 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.497900 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istiod-ca-cert\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.498183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.497933 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-certs\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.498183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.497978 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv497\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-kube-api-access-gv497\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.498183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498007 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-envoy\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.498183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498045 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-socket\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.498183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498046 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:10:29.498183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498073 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-data\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.498183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498108 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-podinfo\") pod \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\" (UID: \"37ce25c4-81a5-4360-a2f0-a4f534ada6ca\") " Sep 10 15:10:29.498531 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498227 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:10:29.498531 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498358 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:10:29.498531 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498403 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:10:29.498691 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498563 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:10:29.498691 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498637 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.498691 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498659 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.498691 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498672 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.498691 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.498685 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.500602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.500577 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-kube-api-access-gv497" (OuterVolumeSpecName: "kube-api-access-gv497") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "kube-api-access-gv497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:10:29.500602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.500591 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:10:29.500748 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.500662 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:10:29.500748 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.500678 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37ce25c4-81a5-4360-a2f0-a4f534ada6ca" (UID: "37ce25c4-81a5-4360-a2f0-a4f534ada6ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:10:29.599311 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.599270 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv497\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-kube-api-access-gv497\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.599311 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.599308 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.599311 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.599317 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.599504 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.599326 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.599504 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.599333 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ce25c4-81a5-4360-a2f0-a4f534ada6ca-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:10:29.614264 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.614236 2560 generic.go:356] "Generic (PLEG): container finished" podID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerID="b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf" exitCode=0 Sep 10 15:10:29.614372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.614286 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" Sep 10 15:10:29.614372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.614284 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" event={"ID":"37ce25c4-81a5-4360-a2f0-a4f534ada6ca","Type":"ContainerDied","Data":"b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf"} Sep 10 15:10:29.614372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.614337 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj" event={"ID":"37ce25c4-81a5-4360-a2f0-a4f534ada6ca","Type":"ContainerDied","Data":"b3037e6d2af98cb9ad24668ef220da594febcb081ac8c61e2a4bdc2db7c38296"} Sep 10 15:10:29.614372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.614358 2560 scope.go:117] "RemoveContainer" containerID="b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf" Sep 10 15:10:29.624207 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.624192 2560 scope.go:117] "RemoveContainer" containerID="1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277" Sep 10 15:10:29.632070 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.632054 2560 scope.go:117] "RemoveContainer" containerID="b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf" Sep 10 15:10:29.632330 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:10:29.632313 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf\": container with ID starting with b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf not found: ID does not exist" containerID="b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf" Sep 10 15:10:29.632381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.632337 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf"} err="failed to get container status \"b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf\": rpc error: code = NotFound desc = could not find container \"b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf\": container with ID starting with b2b41d67c66add032ca2a19ec77a9d5293d5be21fedc6b31887c29013e4523cf not found: ID does not exist" Sep 10 15:10:29.632381 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.632353 2560 scope.go:117] "RemoveContainer" containerID="1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277" Sep 10 15:10:29.632578 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:10:29.632561 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277\": container with ID starting with 1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277 not found: ID does not exist" containerID="1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277" Sep 10 15:10:29.632631 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.632587 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277"} err="failed to get container status \"1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277\": rpc error: code = NotFound desc = could not find container \"1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277\": container with ID starting with 1cfb220ee692a428c42eca0afa2b8ddfaa3faf8619e95d82a633f0504f653277 not found: ID does not exist" Sep 10 15:10:29.637785 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.637761 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj"] Sep 10 15:10:29.642113 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.642092 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5c801c28-openshift-default-688f76bdd-5lqkj"] Sep 10 15:10:29.665102 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:10:29.665077 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" path="/var/lib/kubelet/pods/37ce25c4-81a5-4360-a2f0-a4f534ada6ca/volumes" Sep 10 15:11:18.419161 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.419127 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg"] Sep 10 15:11:18.419751 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.419618 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" Sep 10 15:11:18.419751 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.419632 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" Sep 10 15:11:18.419751 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.419640 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" Sep 10 15:11:18.419751 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.419646 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" Sep 10 15:11:18.419751 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.419731 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" Sep 10 15:11:18.422620 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.422601 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.425341 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.425323 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3e7a73-openshift-default-dockercfg-l8tcm\"" Sep 10 15:11:18.438698 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.438672 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg"] Sep 10 15:11:18.484508 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484474 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshhx\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-kube-api-access-jshhx\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.484677 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484532 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-certs\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.484677 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-data\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.484677 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484593 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-token\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.484677 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484644 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-podinfo\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.484913 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484686 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-envoy\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.484913 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484720 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-credential-socket\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.484913 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484779 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istiod-ca-cert\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.484913 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.484852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-socket\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.585664 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585631 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-certs\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.585664 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-data\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.585937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585688 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-token\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.585937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585709 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-podinfo\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.585937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585735 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-envoy\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.585937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585777 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-credential-socket\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.585937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585860 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istiod-ca-cert\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.585937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585923 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-socket\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.586254 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.585966 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshhx\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-kube-api-access-jshhx\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.586254 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.586097 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-data\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.586254 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.586225 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-credential-socket\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.586450 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.586431 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-certs\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.586520 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.586473 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-socket\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.586729 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.586707 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istiod-ca-cert\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.588145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.588124 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-envoy\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.588336 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.588319 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-podinfo\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.594924 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.594893 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-token\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.595123 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.595104 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshhx\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-kube-api-access-jshhx\") pod \"e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:18.733081 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:18.732992 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:19.014543 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:19.014514 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg"] Sep 10 15:11:19.015844 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:11:19.015818 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d247de9_0cd7_415d_9d77_9e9fba8cccf4.slice/crio-c76ea3a98a61f3e4937da11f635f1b33618e1dc1cd8720491b0c92bc163b871d WatchSource:0}: Error finding container c76ea3a98a61f3e4937da11f635f1b33618e1dc1cd8720491b0c92bc163b871d: Status 404 returned error can't find the container with id c76ea3a98a61f3e4937da11f635f1b33618e1dc1cd8720491b0c92bc163b871d Sep 10 15:11:19.018154 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:19.018121 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:19.018221 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:19.018190 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:19.018259 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:19.018219 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:19.869162 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:19.869124 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" event={"ID":"5d247de9-0cd7-415d-9d77-9e9fba8cccf4","Type":"ContainerStarted","Data":"0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa"} Sep 10 15:11:19.869587 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:19.869176 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" event={"ID":"5d247de9-0cd7-415d-9d77-9e9fba8cccf4","Type":"ContainerStarted","Data":"c76ea3a98a61f3e4937da11f635f1b33618e1dc1cd8720491b0c92bc163b871d"} Sep 10 15:11:19.896439 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:19.896388 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" podStartSLOduration=1.8963727719999999 podStartE2EDuration="1.896372772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:19.892957355 +0000 UTC m=+1418.707540105" watchObservedRunningTime="2026-09-10 15:11:19.896372772 +0000 UTC m=+1418.710955523" Sep 10 15:11:20.733466 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:20.733442 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:20.739261 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:20.739235 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:20.879454 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:20.876643 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:20.880370 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:20.880348 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:45.031408 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:45.031373 2560 generic.go:356] "Generic (PLEG): container finished" podID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerID="0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa" exitCode=0 Sep 10 15:11:45.031851 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:45.031447 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" event={"ID":"5d247de9-0cd7-415d-9d77-9e9fba8cccf4","Type":"ContainerDied","Data":"0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa"} Sep 10 15:11:45.031918 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:45.031902 2560 scope.go:117] "RemoveContainer" containerID="0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa" Sep 10 15:11:45.033136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:45.033107 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:45.033214 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:45.033178 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:45.033214 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:45.033208 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:46.039235 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:46.039188 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" event={"ID":"5d247de9-0cd7-415d-9d77-9e9fba8cccf4","Type":"ContainerStarted","Data":"7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909"} Sep 10 15:11:46.733637 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:46.733604 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:46.734614 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:46.734588 2560 patch_prober.go:28] interesting pod/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:11:46.734732 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:46.734637 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:11:47.589546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.589511 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt"] Sep 10 15:11:47.590325 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.590297 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ce25c4-81a5-4360-a2f0-a4f534ada6ca" containerName="istio-proxy" Sep 10 15:11:47.593102 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.593081 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.597367 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.597342 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d7aa05-openshift-default-dockercfg-jv9x5\"" Sep 10 15:11:47.612905 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.612882 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt"] Sep 10 15:11:47.695823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.695683 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-socket\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.695823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.695744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-envoy\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.695823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.695765 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-data\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.696136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.695836 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-credential-socket\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.696136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.695871 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istiod-ca-cert\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.696136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.695958 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44q4p\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-kube-api-access-44q4p\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.696136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.696002 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-certs\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.696136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.696034 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-token\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.696136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.696129 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-podinfo\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.735532 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.735493 2560 patch_prober.go:28] interesting pod/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:11:47.735868 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.735832 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:11:47.797108 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44q4p\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-kube-api-access-44q4p\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.797316 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797118 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-certs\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.797316 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797140 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-token\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.797316 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797194 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-podinfo\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.797316 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797219 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-socket\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.797316 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797247 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-envoy\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.797609 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-data\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.797609 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797543 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-credential-socket\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.797609 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797572 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istiod-ca-cert\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.798130 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.797754 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-socket\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.798130 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.798010 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-data\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.798130 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.798077 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-credential-socket\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.798424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.798347 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-certs\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.798424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.798384 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istiod-ca-cert\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.799812 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.799769 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-envoy\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.800403 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.800382 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-podinfo\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.805341 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.805301 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-token\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.806217 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.806179 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44q4p\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-kube-api-access-44q4p\") pod \"e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:47.908180 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:47.908143 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:48.204814 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:48.204772 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt"] Sep 10 15:11:48.207662 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:11:48.207635 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fadfa6_9b8d_44ec_90f7_79a7b5279b49.slice/crio-7984d631ab848603417eddd973f13a817116e2eb48394349e8af4ba6c3612f65 WatchSource:0}: Error finding container 7984d631ab848603417eddd973f13a817116e2eb48394349e8af4ba6c3612f65: Status 404 returned error can't find the container with id 7984d631ab848603417eddd973f13a817116e2eb48394349e8af4ba6c3612f65 Sep 10 15:11:48.210458 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:48.210424 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:48.210546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:48.210488 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:48.210546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:48.210517 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:11:48.734002 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:48.733965 2560 patch_prober.go:28] interesting pod/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:11:48.734494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:48.734028 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:11:49.060341 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:49.060243 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" event={"ID":"83fadfa6-9b8d-44ec-90f7-79a7b5279b49","Type":"ContainerStarted","Data":"fa3cc2010830c3e7c91ffa45b475b809edc183fa894093b919a4e1c194fd18af"} Sep 10 15:11:49.060341 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:49.060312 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" event={"ID":"83fadfa6-9b8d-44ec-90f7-79a7b5279b49","Type":"ContainerStarted","Data":"7984d631ab848603417eddd973f13a817116e2eb48394349e8af4ba6c3612f65"} Sep 10 15:11:49.085018 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:49.084969 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" podStartSLOduration=2.084953426 podStartE2EDuration="2.084953426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:49.082167835 +0000 UTC m=+1447.896750587" watchObservedRunningTime="2026-09-10 15:11:49.084953426 +0000 UTC m=+1447.899536177" Sep 10 15:11:49.733606 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:49.733571 2560 patch_prober.go:28] interesting pod/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:11:49.733780 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:49.733622 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:11:49.908701 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:49.908670 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:49.913912 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:49.913889 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:50.069844 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:50.069605 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:50.074441 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:50.074418 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:11:50.733473 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:50.733444 2560 patch_prober.go:28] interesting pod/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 15:11:50.733638 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:50.733497 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 15:11:50.877487 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:50.877452 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:51.737731 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:51.737700 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:11:51.738690 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:11:51.738672 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:12:14.222102 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:14.222070 2560 generic.go:356] "Generic (PLEG): container finished" podID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerID="fa3cc2010830c3e7c91ffa45b475b809edc183fa894093b919a4e1c194fd18af" exitCode=0 Sep 10 15:12:14.222514 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:14.222148 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" event={"ID":"83fadfa6-9b8d-44ec-90f7-79a7b5279b49","Type":"ContainerDied","Data":"fa3cc2010830c3e7c91ffa45b475b809edc183fa894093b919a4e1c194fd18af"} Sep 10 15:12:14.222588 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:14.222574 2560 scope.go:117] "RemoveContainer" containerID="fa3cc2010830c3e7c91ffa45b475b809edc183fa894093b919a4e1c194fd18af" Sep 10 15:12:14.223702 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:14.223671 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:12:14.223822 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:14.223757 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:12:14.223893 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:14.223826 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:12:15.230720 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:15.230687 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" event={"ID":"83fadfa6-9b8d-44ec-90f7-79a7b5279b49","Type":"ContainerStarted","Data":"0f6289f0cab0603394b12bcfcf8ef9e3859345c2766f155483c031d86b7e5e1e"} Sep 10 15:12:15.909035 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:15.908996 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:12:15.910107 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:15.910084 2560 patch_prober.go:28] interesting pod/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 15:12:15.910162 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:15.910133 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 15:12:16.908746 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:16.908708 2560 patch_prober.go:28] interesting pod/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 15:12:16.909143 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:16.908765 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 15:12:17.909494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:17.909458 2560 patch_prober.go:28] interesting pod/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 15:12:17.909909 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:17.909513 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 15:12:18.908782 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:18.908746 2560 patch_prober.go:28] interesting pod/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 15:12:18.909015 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:18.908818 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 15:12:19.908899 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:19.908841 2560 patch_prober.go:28] interesting pod/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 15:12:19.909264 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:19.908984 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 15:12:20.069486 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:20.069441 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:12:20.913093 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:20.913068 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:12:20.913986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:20.913967 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:12:41.511408 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:41.511375 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg"] Sep 10 15:12:41.511888 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:41.511583 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" containerID="cri-o://7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909" gracePeriod=30 Sep 10 15:12:41.768468 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:41.768397 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:12:41.771225 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:41.771203 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:12:46.766214 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.766186 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:12:46.902826 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.902776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-data\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.902991 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.902850 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-certs\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.902991 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.902893 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istiod-ca-cert\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.902991 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.902916 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-envoy\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.902991 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.902971 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-podinfo\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.903240 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.902999 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-credential-socket\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.903240 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903033 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-socket\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.903240 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903045 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:12:46.903240 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903064 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jshhx\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-kube-api-access-jshhx\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.903240 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903147 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-token\") pod \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\" (UID: \"5d247de9-0cd7-415d-9d77-9e9fba8cccf4\") " Sep 10 15:12:46.903240 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903152 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:12:46.903548 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903291 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:12:46.903548 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903314 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:12:46.903548 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903408 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:12:46.903700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903622 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:46.903700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903638 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:46.903700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903647 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:46.903700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903655 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:46.903700 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.903663 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:46.905205 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.905179 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:12:46.905332 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.905293 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:12:46.905592 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.905575 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-kube-api-access-jshhx" (OuterVolumeSpecName: "kube-api-access-jshhx") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "kube-api-access-jshhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:12:46.905645 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:46.905591 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d247de9-0cd7-415d-9d77-9e9fba8cccf4" (UID: "5d247de9-0cd7-415d-9d77-9e9fba8cccf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:12:47.004266 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.004234 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:47.004266 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.004262 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:47.004440 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.004290 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshhx\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-kube-api-access-jshhx\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:47.004440 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.004300 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d247de9-0cd7-415d-9d77-9e9fba8cccf4-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:12:47.400157 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.400119 2560 generic.go:356] "Generic (PLEG): container finished" podID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerID="7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909" exitCode=0 Sep 10 15:12:47.400313 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.400153 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" event={"ID":"5d247de9-0cd7-415d-9d77-9e9fba8cccf4","Type":"ContainerDied","Data":"7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909"} Sep 10 15:12:47.400313 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.400191 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" Sep 10 15:12:47.400313 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.400231 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg" event={"ID":"5d247de9-0cd7-415d-9d77-9e9fba8cccf4","Type":"ContainerDied","Data":"c76ea3a98a61f3e4937da11f635f1b33618e1dc1cd8720491b0c92bc163b871d"} Sep 10 15:12:47.400313 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.400254 2560 scope.go:117] "RemoveContainer" containerID="7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909" Sep 10 15:12:47.410459 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.410440 2560 scope.go:117] "RemoveContainer" containerID="0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa" Sep 10 15:12:47.418508 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.418488 2560 scope.go:117] "RemoveContainer" containerID="7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909" Sep 10 15:12:47.418778 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:12:47.418757 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909\": container with ID starting with 7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909 not found: ID does not exist" containerID="7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909" Sep 10 15:12:47.418879 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.418786 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909"} err="failed to get container status \"7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909\": rpc error: code = NotFound desc = could not find container \"7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909\": container with ID starting with 7efb69f077e7805a5d473ce6b78c14694246191814cbb1f8ea0e361bd4f4e909 not found: ID does not exist" Sep 10 15:12:47.418879 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.418817 2560 scope.go:117] "RemoveContainer" containerID="0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa" Sep 10 15:12:47.419066 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:12:47.419047 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa\": container with ID starting with 0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa not found: ID does not exist" containerID="0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa" Sep 10 15:12:47.419108 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.419074 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa"} err="failed to get container status \"0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa\": rpc error: code = NotFound desc = could not find container \"0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa\": container with ID starting with 0bd2378192588ab99fe906b59f077b17871242d8e25fb802dbfcd7b092dc90fa not found: ID does not exist" Sep 10 15:12:47.422566 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.422546 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg"] Sep 10 15:12:47.430575 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.430553 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3e7a73-openshift-default-bfd9479b4-6cxjg"] Sep 10 15:12:47.666234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:12:47.666145 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" path="/var/lib/kubelet/pods/5d247de9-0cd7-415d-9d77-9e9fba8cccf4/volumes" Sep 10 15:13:04.286362 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:04.286279 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt"] Sep 10 15:13:04.286961 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:04.286502 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" containerID="cri-o://0f6289f0cab0603394b12bcfcf8ef9e3859345c2766f155483c031d86b7e5e1e" gracePeriod=30 Sep 10 15:13:05.915892 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:05.915852 2560 patch_prober.go:28] interesting pod/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 15:13:05.916258 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:05.915908 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 15:13:09.519021 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.518977 2560 generic.go:356] "Generic (PLEG): container finished" podID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerID="0f6289f0cab0603394b12bcfcf8ef9e3859345c2766f155483c031d86b7e5e1e" exitCode=0 Sep 10 15:13:09.519463 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.519087 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" event={"ID":"83fadfa6-9b8d-44ec-90f7-79a7b5279b49","Type":"ContainerDied","Data":"0f6289f0cab0603394b12bcfcf8ef9e3859345c2766f155483c031d86b7e5e1e"} Sep 10 15:13:09.519463 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.519175 2560 scope.go:117] "RemoveContainer" containerID="fa3cc2010830c3e7c91ffa45b475b809edc183fa894093b919a4e1c194fd18af" Sep 10 15:13:09.546143 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.546122 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:13:09.619276 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619250 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-token\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619428 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619329 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44q4p\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-kube-api-access-44q4p\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619428 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619361 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-data\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619428 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619398 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-podinfo\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619573 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619453 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-socket\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619573 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619475 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-credential-socket\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619573 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619503 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istiod-ca-cert\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619573 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619546 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-envoy\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619782 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619570 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-certs\") pod \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\" (UID: \"83fadfa6-9b8d-44ec-90f7-79a7b5279b49\") " Sep 10 15:13:09.619782 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619691 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-data" (OuterVolumeSpecName: "istio-data") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:09.619926 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619869 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:09.619926 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619899 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:09.619926 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.619911 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:09.620077 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.620063 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:09.620132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.620081 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:09.620132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.620096 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:09.620132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.620110 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:09.620290 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.620156 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:09.621581 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.621558 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-token" (OuterVolumeSpecName: "istio-token") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:09.621922 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.621903 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-kube-api-access-44q4p" (OuterVolumeSpecName: "kube-api-access-44q4p") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "kube-api-access-44q4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:09.622022 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.621966 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:09.622104 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.622089 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83fadfa6-9b8d-44ec-90f7-79a7b5279b49" (UID: "83fadfa6-9b8d-44ec-90f7-79a7b5279b49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:13:09.721598 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.721564 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:09.721598 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.721591 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44q4p\" (UniqueName: \"kubernetes.io/projected/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-kube-api-access-44q4p\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:09.721811 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.721605 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:09.721811 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.721618 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:09.721811 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:09.721630 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fadfa6-9b8d-44ec-90f7-79a7b5279b49-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:10.525434 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:10.525408 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" Sep 10 15:13:10.525862 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:10.525402 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt" event={"ID":"83fadfa6-9b8d-44ec-90f7-79a7b5279b49","Type":"ContainerDied","Data":"7984d631ab848603417eddd973f13a817116e2eb48394349e8af4ba6c3612f65"} Sep 10 15:13:10.525862 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:10.525538 2560 scope.go:117] "RemoveContainer" containerID="0f6289f0cab0603394b12bcfcf8ef9e3859345c2766f155483c031d86b7e5e1e" Sep 10 15:13:10.543360 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:10.543334 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt"] Sep 10 15:13:10.546393 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:10.546368 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d7aa05-openshift-default-58cc4db5c8-hx9nt"] Sep 10 15:13:11.665097 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:11.665064 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" path="/var/lib/kubelet/pods/83fadfa6-9b8d-44ec-90f7-79a7b5279b49/volumes" Sep 10 15:13:15.452632 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.452600 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf"] Sep 10 15:13:15.453087 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.452817 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" podUID="e3449fdb-c3de-4a7d-83e0-f0dd60a17903" containerName="maas-api" containerID="cri-o://8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d" gracePeriod=30 Sep 10 15:13:15.697507 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.697485 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:13:15.779154 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.779076 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-openshift-service-ca\") pod \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " Sep 10 15:13:15.779154 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.779137 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls\") pod \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " Sep 10 15:13:15.779393 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.779190 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-metrics-certs\") pod \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " Sep 10 15:13:15.779393 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.779329 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssk7c\" (UniqueName: \"kubernetes.io/projected/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-kube-api-access-ssk7c\") pod \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\" (UID: \"e3449fdb-c3de-4a7d-83e0-f0dd60a17903\") " Sep 10 15:13:15.779564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.779458 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3449fdb-c3de-4a7d-83e0-f0dd60a17903" (UID: "e3449fdb-c3de-4a7d-83e0-f0dd60a17903"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:15.779791 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.779762 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:15.781222 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.781189 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-kube-api-access-ssk7c" (OuterVolumeSpecName: "kube-api-access-ssk7c") pod "e3449fdb-c3de-4a7d-83e0-f0dd60a17903" (UID: "e3449fdb-c3de-4a7d-83e0-f0dd60a17903"). InnerVolumeSpecName "kube-api-access-ssk7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:15.781315 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.781223 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3449fdb-c3de-4a7d-83e0-f0dd60a17903" (UID: "e3449fdb-c3de-4a7d-83e0-f0dd60a17903"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:13:15.781315 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.781252 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3449fdb-c3de-4a7d-83e0-f0dd60a17903" (UID: "e3449fdb-c3de-4a7d-83e0-f0dd60a17903"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:13:15.880544 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.880512 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:15.880544 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.880541 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:15.880544 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:15.880550 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssk7c\" (UniqueName: \"kubernetes.io/projected/e3449fdb-c3de-4a7d-83e0-f0dd60a17903-kube-api-access-ssk7c\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:16.560535 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.560499 2560 generic.go:356] "Generic (PLEG): container finished" podID="e3449fdb-c3de-4a7d-83e0-f0dd60a17903" containerID="8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d" exitCode=0 Sep 10 15:13:16.560535 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.560541 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" Sep 10 15:13:16.561099 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.560576 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" event={"ID":"e3449fdb-c3de-4a7d-83e0-f0dd60a17903","Type":"ContainerDied","Data":"8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d"} Sep 10 15:13:16.561099 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.560629 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf" event={"ID":"e3449fdb-c3de-4a7d-83e0-f0dd60a17903","Type":"ContainerDied","Data":"ec8b91a86aed066eda46d7f6f5fbf9b1ff829712f80f0acbff28b61f757ec0c8"} Sep 10 15:13:16.561099 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.560650 2560 scope.go:117] "RemoveContainer" containerID="8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d" Sep 10 15:13:16.571706 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.571688 2560 scope.go:117] "RemoveContainer" containerID="8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d" Sep 10 15:13:16.571998 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:13:16.571975 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d\": container with ID starting with 8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d not found: ID does not exist" containerID="8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d" Sep 10 15:13:16.572103 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.572003 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d"} err="failed to get container status \"8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d\": rpc error: code = NotFound desc = could not find container \"8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d\": container with ID starting with 8465f456c1d9c63a9522dd081787393e34d538db1997a883f2ea3ae002b4134d not found: ID does not exist" Sep 10 15:13:16.586783 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.586755 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf"] Sep 10 15:13:16.591104 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:16.591080 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0a9de4-66fc6ddc7-qt2sf"] Sep 10 15:13:17.668397 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:17.668355 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3449fdb-c3de-4a7d-83e0-f0dd60a17903" path="/var/lib/kubelet/pods/e3449fdb-c3de-4a7d-83e0-f0dd60a17903/volumes" Sep 10 15:13:21.910038 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:21.910003 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd"] Sep 10 15:13:21.910500 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:21.910214 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" containerID="cri-o://52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c" gracePeriod=30 Sep 10 15:13:27.156472 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.156451 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:13:27.293527 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293444 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad40d8-cdfb-4d49-9604-0fdab9515172-istiod-ca-cert\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.293527 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293499 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-envoy\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.293527 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293518 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-token\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.293854 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293546 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-socket\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.293854 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293573 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-credential-socket\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.293854 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293604 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-certs\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.293854 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293632 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-data\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.293854 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293760 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lggg\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-kube-api-access-4lggg\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.293854 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293829 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-podinfo\") pod \"ecad40d8-cdfb-4d49-9604-0fdab9515172\" (UID: \"ecad40d8-cdfb-4d49-9604-0fdab9515172\") " Sep 10 15:13:27.294273 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293867 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecad40d8-cdfb-4d49-9604-0fdab9515172-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:27.294273 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293954 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-data" (OuterVolumeSpecName: "istio-data") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:27.294273 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.293965 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:27.294273 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.294233 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.294273 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.294253 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad40d8-cdfb-4d49-9604-0fdab9515172-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.294273 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.294268 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.294546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.294303 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:27.294546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.294375 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:27.296236 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.296213 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:27.296303 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.296233 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-token" (OuterVolumeSpecName: "istio-token") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:27.296388 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.296375 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:13:27.296530 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.296503 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-kube-api-access-4lggg" (OuterVolumeSpecName: "kube-api-access-4lggg") pod "ecad40d8-cdfb-4d49-9604-0fdab9515172" (UID: "ecad40d8-cdfb-4d49-9604-0fdab9515172"). InnerVolumeSpecName "kube-api-access-4lggg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:27.395685 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.395648 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lggg\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-kube-api-access-4lggg\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.395685 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.395679 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.395685 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.395692 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.395937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.395703 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad40d8-cdfb-4d49-9604-0fdab9515172-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.395937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.395713 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.395937 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.395724 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad40d8-cdfb-4d49-9604-0fdab9515172-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:27.623501 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.623471 2560 generic.go:356] "Generic (PLEG): container finished" podID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerID="52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c" exitCode=0 Sep 10 15:13:27.623654 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.623519 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" Sep 10 15:13:27.623654 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.623527 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" event={"ID":"ecad40d8-cdfb-4d49-9604-0fdab9515172","Type":"ContainerDied","Data":"52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c"} Sep 10 15:13:27.623654 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.623568 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd" event={"ID":"ecad40d8-cdfb-4d49-9604-0fdab9515172","Type":"ContainerDied","Data":"f22676c271bfc8c68d8f834f4206791f481a40b4ac498b8a35fc74df6f4729bb"} Sep 10 15:13:27.623654 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.623589 2560 scope.go:117] "RemoveContainer" containerID="52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c" Sep 10 15:13:27.632754 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.632733 2560 scope.go:117] "RemoveContainer" containerID="76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7" Sep 10 15:13:27.640548 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.640530 2560 scope.go:117] "RemoveContainer" containerID="52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c" Sep 10 15:13:27.640828 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:13:27.640807 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c\": container with ID starting with 52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c not found: ID does not exist" containerID="52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c" Sep 10 15:13:27.640918 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.640835 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c"} err="failed to get container status \"52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c\": rpc error: code = NotFound desc = could not find container \"52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c\": container with ID starting with 52291bee7105a8ecd54b63a7b07bc2ade0b2e356fc95fba8b509cd42fa0c8d8c not found: ID does not exist" Sep 10 15:13:27.640918 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.640853 2560 scope.go:117] "RemoveContainer" containerID="76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7" Sep 10 15:13:27.641111 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:13:27.641094 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7\": container with ID starting with 76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7 not found: ID does not exist" containerID="76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7" Sep 10 15:13:27.641152 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.641118 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7"} err="failed to get container status \"76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7\": rpc error: code = NotFound desc = could not find container \"76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7\": container with ID starting with 76f0b77f14f278b4bb8db58b18cad6d3e2ec1aa22a88b30e3be9e941d3ac5cb7 not found: ID does not exist" Sep 10 15:13:27.645698 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.645675 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd"] Sep 10 15:13:27.649513 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.649493 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0a9de4-openshift-default-8458f85fb8-ms6wd"] Sep 10 15:13:27.665177 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:27.665156 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" path="/var/lib/kubelet/pods/ecad40d8-cdfb-4d49-9604-0fdab9515172/volumes" Sep 10 15:13:39.701489 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:39.701457 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk"] Sep 10 15:13:39.701977 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:39.701655 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" containerID="cri-o://1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907" gracePeriod=30 Sep 10 15:13:44.947815 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:44.947782 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:13:45.063298 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063210 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d602a4c-1d84-438b-89dc-09d049ea4499-istiod-ca-cert\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063298 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063268 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-podinfo\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063298 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063297 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-data\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063588 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063446 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-envoy\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063588 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063476 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-socket\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063588 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063501 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-token\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063588 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063524 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-credential-socket\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063830 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063590 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzwz2\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-kube-api-access-pzwz2\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063830 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063615 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:45.063830 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063645 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-certs\") pod \"8d602a4c-1d84-438b-89dc-09d049ea4499\" (UID: \"8d602a4c-1d84-438b-89dc-09d049ea4499\") " Sep 10 15:13:45.063830 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063723 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:45.063830 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063634 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d602a4c-1d84-438b-89dc-09d049ea4499-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:13:45.064114 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063904 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:45.064114 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.063963 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:45.064114 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.064102 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d602a4c-1d84-438b-89dc-09d049ea4499-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.064114 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.064119 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.064331 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.064132 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.064331 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.064144 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.064331 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.064155 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.065754 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.065719 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:13:45.065898 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.065775 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-kube-api-access-pzwz2" (OuterVolumeSpecName: "kube-api-access-pzwz2") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "kube-api-access-pzwz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:45.065985 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.065956 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:13:45.066045 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.066011 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d602a4c-1d84-438b-89dc-09d049ea4499" (UID: "8d602a4c-1d84-438b-89dc-09d049ea4499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:13:45.165511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.165466 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.165511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.165507 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.165511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.165515 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.165511 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.165523 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzwz2\" (UniqueName: \"kubernetes.io/projected/8d602a4c-1d84-438b-89dc-09d049ea4499-kube-api-access-pzwz2\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:13:45.723780 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.723749 2560 generic.go:356] "Generic (PLEG): container finished" podID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerID="1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907" exitCode=0 Sep 10 15:13:45.723934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.723785 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" event={"ID":"8d602a4c-1d84-438b-89dc-09d049ea4499","Type":"ContainerDied","Data":"1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907"} Sep 10 15:13:45.723934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.723811 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" Sep 10 15:13:45.723934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.723847 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk" event={"ID":"8d602a4c-1d84-438b-89dc-09d049ea4499","Type":"ContainerDied","Data":"e021a9f6e6a037f54b9641ef9772b55746f3d408c58a43d9729c362aa01ef8e5"} Sep 10 15:13:45.723934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.723869 2560 scope.go:117] "RemoveContainer" containerID="1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907" Sep 10 15:13:45.732767 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.732746 2560 scope.go:117] "RemoveContainer" containerID="dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b" Sep 10 15:13:45.741019 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.740995 2560 scope.go:117] "RemoveContainer" containerID="1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907" Sep 10 15:13:45.741312 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:13:45.741290 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907\": container with ID starting with 1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907 not found: ID does not exist" containerID="1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907" Sep 10 15:13:45.741405 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.741321 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907"} err="failed to get container status \"1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907\": rpc error: code = NotFound desc = could not find container \"1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907\": container with ID starting with 1e6b680472bf721e62bdef3b9f5fd43552380f590785c341ff36c017c3f0e907 not found: ID does not exist" Sep 10 15:13:45.741405 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.741339 2560 scope.go:117] "RemoveContainer" containerID="dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b" Sep 10 15:13:45.741632 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:13:45.741607 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b\": container with ID starting with dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b not found: ID does not exist" containerID="dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b" Sep 10 15:13:45.741684 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.741629 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b"} err="failed to get container status \"dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b\": rpc error: code = NotFound desc = could not find container \"dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b\": container with ID starting with dbb4e992023e9e68ab2e621dcc8c509b7eefa27c26f231118b6887a94e50042b not found: ID does not exist" Sep 10 15:13:45.743664 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.743643 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk"] Sep 10 15:13:45.745427 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:45.745407 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1f985c-openshift-default-7d86f78884-p69lk"] Sep 10 15:13:47.665269 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:47.665234 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" path="/var/lib/kubelet/pods/8d602a4c-1d84-438b-89dc-09d049ea4499/volumes" Sep 10 15:13:59.267261 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267217 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f"] Sep 10 15:13:59.267757 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267737 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" Sep 10 15:13:59.267823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267759 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" Sep 10 15:13:59.267823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267767 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" Sep 10 15:13:59.267823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267773 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" Sep 10 15:13:59.267823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267781 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" Sep 10 15:13:59.267823 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267787 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267828 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267836 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267843 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267849 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267858 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267864 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267874 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3449fdb-c3de-4a7d-83e0-f0dd60a17903" containerName="maas-api" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267882 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3449fdb-c3de-4a7d-83e0-f0dd60a17903" containerName="maas-api" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267896 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267901 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267909 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267914 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267983 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" Sep 10 15:13:59.267986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267990 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" Sep 10 15:13:59.268745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.267998 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3449fdb-c3de-4a7d-83e0-f0dd60a17903" containerName="maas-api" Sep 10 15:13:59.268745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.268003 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" Sep 10 15:13:59.268745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.268008 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fadfa6-9b8d-44ec-90f7-79a7b5279b49" containerName="istio-proxy" Sep 10 15:13:59.268745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.268015 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d247de9-0cd7-415d-9d77-9e9fba8cccf4" containerName="istio-proxy" Sep 10 15:13:59.268745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.268020 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" Sep 10 15:13:59.268745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.268026 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecad40d8-cdfb-4d49-9604-0fdab9515172" containerName="istio-proxy" Sep 10 15:13:59.271737 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.271715 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.274132 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.274108 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-13ee06-openshift-default-dockercfg-9l9pk\"" Sep 10 15:13:59.283444 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.283420 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f"] Sep 10 15:13:59.388992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.388953 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-token\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.389289 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.389248 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-podinfo\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.389423 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.389296 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-certs\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.389423 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.389335 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-data\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.389423 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.389378 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-envoy\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.389602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.389455 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-credential-socket\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.389602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.389505 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzp5\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-kube-api-access-fpzp5\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.389602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.389532 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istiod-ca-cert\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.389602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.389574 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-socket\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.490541 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490500 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-credential-socket\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.490726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490554 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzp5\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-kube-api-access-fpzp5\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.490726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istiod-ca-cert\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.490726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490614 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-socket\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.490726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490687 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-token\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.490726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490721 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-podinfo\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.491031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490770 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-certs\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.491031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490827 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-data\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.491031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-envoy\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.491031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490926 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-credential-socket\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.491031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.490990 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-socket\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.491293 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.491070 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-certs\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.491293 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.491126 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-data\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.491571 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.491556 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istiod-ca-cert\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.493036 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.493010 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-envoy\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.493142 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.493098 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-podinfo\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.499360 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.499335 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzp5\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-kube-api-access-fpzp5\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.500291 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.500273 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-token\") pod \"e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.585319 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.585237 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:13:59.857593 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.857570 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f"] Sep 10 15:13:59.860006 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:13:59.859980 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd1d987_ac83_4ef7_9da9_454aa9a1aeb0.slice/crio-37a749c9c7e15f989bb057efc6962146cf556d000bbcbdb39471601147a388b6 WatchSource:0}: Error finding container 37a749c9c7e15f989bb057efc6962146cf556d000bbcbdb39471601147a388b6: Status 404 returned error can't find the container with id 37a749c9c7e15f989bb057efc6962146cf556d000bbcbdb39471601147a388b6 Sep 10 15:13:59.862526 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.862490 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:13:59.862616 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.862556 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:13:59.862616 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:13:59.862586 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:00.798128 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:00.798089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" event={"ID":"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0","Type":"ContainerStarted","Data":"0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde"} Sep 10 15:14:00.798550 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:00.798165 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" event={"ID":"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0","Type":"ContainerStarted","Data":"37a749c9c7e15f989bb057efc6962146cf556d000bbcbdb39471601147a388b6"} Sep 10 15:14:00.820486 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:00.820436 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" podStartSLOduration=1.8204207000000001 podStartE2EDuration="1.8204207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:14:00.818263187 +0000 UTC m=+1579.632845938" watchObservedRunningTime="2026-09-10 15:14:00.8204207 +0000 UTC m=+1579.635003499" Sep 10 15:14:01.585825 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:01.585765 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:14:01.590969 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:01.590939 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:14:01.802882 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:01.802852 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:14:01.803718 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:01.803700 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:14:03.938461 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:03.938430 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv"] Sep 10 15:14:03.939097 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:03.939078 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d602a4c-1d84-438b-89dc-09d049ea4499" containerName="istio-proxy" Sep 10 15:14:03.941396 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:03.941377 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:03.943723 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:03.943705 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-13ee06-serving-cert\"" Sep 10 15:14:03.957778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:03.957753 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv"] Sep 10 15:14:04.035128 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.035091 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.035309 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.035157 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-metrics-certs\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.035309 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.035269 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.035397 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.035376 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxpz\" (UniqueName: \"kubernetes.io/projected/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-kube-api-access-fzxpz\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.136152 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.136115 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-metrics-certs\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.136328 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.136164 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.136328 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.136210 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxpz\" (UniqueName: \"kubernetes.io/projected/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-kube-api-access-fzxpz\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.136328 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.136246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.136443 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:14:04.136358 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-serving-cert: secret "maas-api-e2e-ipp-a-13ee06-serving-cert" not found Sep 10 15:14:04.136443 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:14:04.136423 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls podName:3256b1ff-16f6-453a-9476-7b61d5ec4ffb nodeName:}" failed. No retries permitted until 2026-09-10 15:14:04.636403651 +0000 UTC m=+1583.450986395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls") pod "maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" (UID: "3256b1ff-16f6-453a-9476-7b61d5ec4ffb") : secret "maas-api-e2e-ipp-a-13ee06-serving-cert" not found Sep 10 15:14:04.136902 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.136881 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.138635 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.138611 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-metrics-certs\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.147191 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.147165 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxpz\" (UniqueName: \"kubernetes.io/projected/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-kube-api-access-fzxpz\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.642063 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.642026 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.644559 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.644516 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:04.857194 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:04.857163 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:05.129344 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:05.129313 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv"] Sep 10 15:14:05.132674 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:14:05.132650 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3256b1ff_16f6_453a_9476_7b61d5ec4ffb.slice/crio-c13a635efe03e2fe3b1d277e5ae1846e12a1ae71b52d8de6f0c354271ef037b5 WatchSource:0}: Error finding container c13a635efe03e2fe3b1d277e5ae1846e12a1ae71b52d8de6f0c354271ef037b5: Status 404 returned error can't find the container with id c13a635efe03e2fe3b1d277e5ae1846e12a1ae71b52d8de6f0c354271ef037b5 Sep 10 15:14:05.133963 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:05.133944 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:14:05.822906 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:05.822860 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" event={"ID":"3256b1ff-16f6-453a-9476-7b61d5ec4ffb","Type":"ContainerStarted","Data":"8a6c28f3d371b5697e36c6d7b7592c5c9919337adb3437f7d0fdb4f025051969"} Sep 10 15:14:05.823082 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:05.822933 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:05.823082 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:05.822958 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" event={"ID":"3256b1ff-16f6-453a-9476-7b61d5ec4ffb","Type":"ContainerStarted","Data":"c13a635efe03e2fe3b1d277e5ae1846e12a1ae71b52d8de6f0c354271ef037b5"} Sep 10 15:14:05.846312 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:05.846257 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" podStartSLOduration=2.351504611 podStartE2EDuration="2.846238532s" totalImagesPullingTime="494.733921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:14:05.844509679 +0000 UTC m=+1584.659092432" watchObservedRunningTime="2026-09-10 15:14:05.846238532 +0000 UTC m=+1584.660821282" Sep 10 15:14:11.832908 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:11.832880 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:14:17.844003 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:17.843956 2560 patch_prober.go:28] interesting pod/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:14:17.844463 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:17.844032 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:14:18.902167 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:18.902137 2560 generic.go:356] "Generic (PLEG): container finished" podID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerID="0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde" exitCode=0 Sep 10 15:14:18.902577 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:18.902193 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" event={"ID":"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0","Type":"ContainerDied","Data":"0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde"} Sep 10 15:14:18.902697 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:18.902679 2560 scope.go:117] "RemoveContainer" containerID="0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde" Sep 10 15:14:18.903873 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:18.903844 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:18.903952 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:18.903922 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:18.903993 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:18.903955 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:19.908822 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:19.908773 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" event={"ID":"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0","Type":"ContainerStarted","Data":"c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe"} Sep 10 15:14:20.585940 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:20.585846 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:14:20.586084 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:20.586053 2560 patch_prober.go:28] interesting pod/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 15:14:20.586128 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:20.586102 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 15:14:21.585659 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:21.585625 2560 patch_prober.go:28] interesting pod/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 15:14:21.586071 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:21.585678 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 15:14:22.585598 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:22.585565 2560 patch_prober.go:28] interesting pod/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 15:14:22.585767 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:22.585629 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 15:14:23.586597 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:23.586560 2560 patch_prober.go:28] interesting pod/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 15:14:23.587019 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:23.586620 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 15:14:24.585991 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:24.585953 2560 patch_prober.go:28] interesting pod/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 10 15:14:24.586159 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:24.586017 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 10 15:14:25.588904 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:25.588874 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:14:25.589267 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:25.589218 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:14:25.589897 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:25.589879 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:14:28.638630 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.638594 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm"] Sep 10 15:14:28.643124 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.643098 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.645676 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.645641 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6333de-openshift-default-dockercfg-xhjd5\"" Sep 10 15:14:28.655096 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.655074 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm"] Sep 10 15:14:28.784672 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-podinfo\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.784883 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784707 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140d701a-71b6-42d8-804c-4048eadbbdfa-istiod-ca-cert\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.784883 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784731 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-token\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.784883 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784750 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-credential-socket\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.784883 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784827 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-data\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.785127 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784884 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-envoy\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.785127 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hmzz\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-kube-api-access-2hmzz\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.785127 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784958 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-certs\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.785127 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.784992 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-socket\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886284 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886247 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-credential-socket\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886466 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886305 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-data\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886466 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886423 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-envoy\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886553 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886542 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hmzz\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-kube-api-access-2hmzz\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886593 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886573 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-certs\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886650 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886592 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-socket\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886699 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886686 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-data\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886749 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886703 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-credential-socket\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886749 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886738 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-podinfo\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886871 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886843 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140d701a-71b6-42d8-804c-4048eadbbdfa-istiod-ca-cert\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886936 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886871 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-token\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.886991 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.886939 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-certs\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.887046 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.887025 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-socket\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.887475 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.887453 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140d701a-71b6-42d8-804c-4048eadbbdfa-istiod-ca-cert\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.888756 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.888705 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-envoy\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.889168 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.889151 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-podinfo\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.897259 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.897234 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hmzz\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-kube-api-access-2hmzz\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.897427 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.897408 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-token\") pod \"e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:28.955323 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:28.955291 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:29.236074 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:29.235856 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm"] Sep 10 15:14:29.245321 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:14:29.245277 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140d701a_71b6_42d8_804c_4048eadbbdfa.slice/crio-c08a9a845be66be616cbb817197723ee18d0dc945256093c5f0c6f7a4a4fb4f0 WatchSource:0}: Error finding container c08a9a845be66be616cbb817197723ee18d0dc945256093c5f0c6f7a4a4fb4f0: Status 404 returned error can't find the container with id c08a9a845be66be616cbb817197723ee18d0dc945256093c5f0c6f7a4a4fb4f0 Sep 10 15:14:29.247942 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:29.247895 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:29.248028 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:29.247978 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:29.248028 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:29.248015 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:29.967232 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:29.967192 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" event={"ID":"140d701a-71b6-42d8-804c-4048eadbbdfa","Type":"ContainerStarted","Data":"3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3"} Sep 10 15:14:29.967562 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:29.967254 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" event={"ID":"140d701a-71b6-42d8-804c-4048eadbbdfa","Type":"ContainerStarted","Data":"c08a9a845be66be616cbb817197723ee18d0dc945256093c5f0c6f7a4a4fb4f0"} Sep 10 15:14:29.989456 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:29.989403 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" podStartSLOduration=1.989387928 podStartE2EDuration="1.989387928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:14:29.987084821 +0000 UTC m=+1608.801667596" watchObservedRunningTime="2026-09-10 15:14:29.989387928 +0000 UTC m=+1608.803970679" Sep 10 15:14:30.955750 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:30.955712 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:30.960533 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:30.960511 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:30.973375 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:30.973355 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:30.974031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:30.974016 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:50.072837 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:50.072729 2560 generic.go:356] "Generic (PLEG): container finished" podID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerID="3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3" exitCode=0 Sep 10 15:14:50.072837 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:50.072822 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" event={"ID":"140d701a-71b6-42d8-804c-4048eadbbdfa","Type":"ContainerDied","Data":"3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3"} Sep 10 15:14:50.073255 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:50.073238 2560 scope.go:117] "RemoveContainer" containerID="3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3" Sep 10 15:14:50.074398 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:50.074364 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:50.074448 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:50.074438 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:50.074488 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:50.074471 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:14:51.081812 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:51.081774 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" event={"ID":"140d701a-71b6-42d8-804c-4048eadbbdfa","Type":"ContainerStarted","Data":"b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258"} Sep 10 15:14:51.956621 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:51.956587 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:51.956923 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:51.956633 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 15:14:51.957079 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:51.957056 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 15:14:52.956508 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:52.956468 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 15:14:52.957121 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:52.957089 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 15:14:53.956033 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:53.955995 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 15:14:53.956241 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:53.956062 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 15:14:54.956188 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:54.956150 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 15:14:54.956746 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:54.956716 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 15:14:55.956606 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:55.956567 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 15:14:55.957128 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:55.956636 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 15:14:56.959842 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:56.959813 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:56.960226 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:56.960100 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:14:56.960791 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:14:56.960775 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:15:43.429830 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.429784 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f"] Sep 10 15:15:43.432740 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.432710 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.435911 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.435888 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-06cc0e-openshift-default-dockercfg-wbs7v\"" Sep 10 15:15:43.446223 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.446198 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f"] Sep 10 15:15:43.501564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501542 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-podinfo\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.501701 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501572 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-certs\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.501701 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501623 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-data\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.501701 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501669 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vq9\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-kube-api-access-v8vq9\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.501701 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793d983-6a7d-42fd-9451-fe05e139bd66-istiod-ca-cert\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.501933 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501730 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-token\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.501933 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501763 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-socket\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.501933 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501873 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-envoy\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.501933 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.501921 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-credential-socket\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603335 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603301 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-podinfo\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603343 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-certs\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603385 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-data\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vq9\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-kube-api-access-v8vq9\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793d983-6a7d-42fd-9451-fe05e139bd66-istiod-ca-cert\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603827 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603719 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-token\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603827 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603759 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-socket\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603947 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603845 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-envoy\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603947 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603887 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-credential-socket\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.603947 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.603908 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-certs\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.604111 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.604078 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-data\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.604259 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.604236 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-credential-socket\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.604351 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.604328 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-socket\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.604410 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.604378 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793d983-6a7d-42fd-9451-fe05e139bd66-istiod-ca-cert\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.605952 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.605923 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-envoy\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.606048 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.605969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-podinfo\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.611623 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.611603 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-token\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.611771 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.611755 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vq9\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-kube-api-access-v8vq9\") pod \"e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:43.745910 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:43.745835 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:44.018491 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:44.018461 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f"] Sep 10 15:15:44.020066 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:15:44.020037 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb793d983_6a7d_42fd_9451_fe05e139bd66.slice/crio-8a7256e9505a176f1be53c9d8b2925da4265bdf364cb56b9b64ff213d71c220d WatchSource:0}: Error finding container 8a7256e9505a176f1be53c9d8b2925da4265bdf364cb56b9b64ff213d71c220d: Status 404 returned error can't find the container with id 8a7256e9505a176f1be53c9d8b2925da4265bdf364cb56b9b64ff213d71c220d Sep 10 15:15:44.022408 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:44.022373 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:15:44.022494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:44.022439 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:15:44.022494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:44.022467 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:15:44.381246 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:44.381208 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" event={"ID":"b793d983-6a7d-42fd-9451-fe05e139bd66","Type":"ContainerStarted","Data":"0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e"} Sep 10 15:15:44.381424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:44.381294 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" event={"ID":"b793d983-6a7d-42fd-9451-fe05e139bd66","Type":"ContainerStarted","Data":"8a7256e9505a176f1be53c9d8b2925da4265bdf364cb56b9b64ff213d71c220d"} Sep 10 15:15:44.402074 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:44.402030 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" podStartSLOduration=1.402014345 podStartE2EDuration="1.402014345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:44.399764052 +0000 UTC m=+1683.214346805" watchObservedRunningTime="2026-09-10 15:15:44.402014345 +0000 UTC m=+1683.216597160" Sep 10 15:15:44.746962 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:44.746890 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:45.751891 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:45.751864 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:46.392013 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:46.391986 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:15:46.392840 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:15:46.392814 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:16:07.509637 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:07.509548 2560 generic.go:356] "Generic (PLEG): container finished" podID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerID="0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e" exitCode=0 Sep 10 15:16:07.509637 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:07.509624 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" event={"ID":"b793d983-6a7d-42fd-9451-fe05e139bd66","Type":"ContainerDied","Data":"0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e"} Sep 10 15:16:07.510076 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:07.510066 2560 scope.go:117] "RemoveContainer" containerID="0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e" Sep 10 15:16:07.511259 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:07.511228 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:16:07.511386 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:07.511311 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:16:07.511386 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:07.511355 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 15:16:08.517085 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:08.517052 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" event={"ID":"b793d983-6a7d-42fd-9451-fe05e139bd66","Type":"ContainerStarted","Data":"c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c"} Sep 10 15:16:08.746155 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:08.746119 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:16:08.746365 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:08.746343 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 15:16:08.746416 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:08.746390 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 15:16:09.746345 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:09.746305 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 15:16:09.746717 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:09.746401 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 15:16:10.746845 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:10.746813 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 15:16:10.747210 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:10.746865 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 15:16:11.746703 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:11.746667 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 15:16:11.746906 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:11.746721 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 15:16:12.747142 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:12.747105 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 15:16:12.747614 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:12.747176 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 15:16:13.750366 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:13.750337 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:16:13.750762 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:13.750742 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:16:13.751349 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:13.751330 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:16:24.566689 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:24.566658 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f"] Sep 10 15:16:24.567098 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:24.566883 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" containerID="cri-o://c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c" gracePeriod=30 Sep 10 15:16:28.752923 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:28.752893 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 10 15:16:28.753310 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:28.752958 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 15:16:29.818616 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.818591 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:16:29.980861 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.980828 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-podinfo\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.980889 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-socket\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.980985 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8vq9\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-kube-api-access-v8vq9\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981027 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-credential-socket\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981053 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793d983-6a7d-42fd-9451-fe05e139bd66-istiod-ca-cert\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981086 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-data\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981120 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-certs\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981141 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-token\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981222 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-envoy\") pod \"b793d983-6a7d-42fd-9451-fe05e139bd66\" (UID: \"b793d983-6a7d-42fd-9451-fe05e139bd66\") " Sep 10 15:16:29.981305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981114 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:16:29.981606 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981271 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:16:29.981606 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981384 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:16:29.981606 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981387 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b793d983-6a7d-42fd-9451-fe05e139bd66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:16:29.981778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981626 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-data" (OuterVolumeSpecName: "istio-data") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:16:29.981778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981700 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:29.981778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981723 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:29.981778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981737 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793d983-6a7d-42fd-9451-fe05e139bd66-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:29.981778 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.981752 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:29.983002 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.982974 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:16:29.983138 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.983087 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-kube-api-access-v8vq9" (OuterVolumeSpecName: "kube-api-access-v8vq9") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "kube-api-access-v8vq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:16:29.983305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.983282 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-token" (OuterVolumeSpecName: "istio-token") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:16:29.983305 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:29.983287 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b793d983-6a7d-42fd-9451-fe05e139bd66" (UID: "b793d983-6a7d-42fd-9451-fe05e139bd66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:16:30.082561 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.082528 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:30.082561 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.082554 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vq9\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-kube-api-access-v8vq9\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:30.082561 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.082565 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:30.082777 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.082574 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:30.082777 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.082582 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793d983-6a7d-42fd-9451-fe05e139bd66-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:30.631852 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.631794 2560 generic.go:356] "Generic (PLEG): container finished" podID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerID="c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c" exitCode=0 Sep 10 15:16:30.631852 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.631856 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" Sep 10 15:16:30.632056 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.631901 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" event={"ID":"b793d983-6a7d-42fd-9451-fe05e139bd66","Type":"ContainerDied","Data":"c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c"} Sep 10 15:16:30.632056 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.631960 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f" event={"ID":"b793d983-6a7d-42fd-9451-fe05e139bd66","Type":"ContainerDied","Data":"8a7256e9505a176f1be53c9d8b2925da4265bdf364cb56b9b64ff213d71c220d"} Sep 10 15:16:30.632056 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.631990 2560 scope.go:117] "RemoveContainer" containerID="c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c" Sep 10 15:16:30.641336 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.641320 2560 scope.go:117] "RemoveContainer" containerID="0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e" Sep 10 15:16:30.649057 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.649037 2560 scope.go:117] "RemoveContainer" containerID="c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c" Sep 10 15:16:30.649311 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:16:30.649294 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c\": container with ID starting with c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c not found: ID does not exist" containerID="c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c" Sep 10 15:16:30.649371 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.649317 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c"} err="failed to get container status \"c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c\": rpc error: code = NotFound desc = could not find container \"c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c\": container with ID starting with c1a65bdc1fc9dd671be6207b212b5c8b6db0a15efbf2aa85e80f7f494beab98c not found: ID does not exist" Sep 10 15:16:30.649371 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.649333 2560 scope.go:117] "RemoveContainer" containerID="0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e" Sep 10 15:16:30.649599 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:16:30.649584 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e\": container with ID starting with 0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e not found: ID does not exist" containerID="0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e" Sep 10 15:16:30.649645 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.649603 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e"} err="failed to get container status \"0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e\": rpc error: code = NotFound desc = could not find container \"0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e\": container with ID starting with 0da6d8cf0ed28af052871d804a37d38664132ee50786b7dba94e6955af70307e not found: ID does not exist" Sep 10 15:16:30.653361 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.653338 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f"] Sep 10 15:16:30.657183 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:30.657164 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-06cc0e-openshift-default-f68764d54-5469f"] Sep 10 15:16:31.663869 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:31.663836 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" path="/var/lib/kubelet/pods/b793d983-6a7d-42fd-9451-fe05e139bd66/volumes" Sep 10 15:16:43.539103 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.539069 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv"] Sep 10 15:16:43.539564 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.539338 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" podUID="3256b1ff-16f6-453a-9476-7b61d5ec4ffb" containerName="maas-api" containerID="cri-o://8a6c28f3d371b5697e36c6d7b7592c5c9919337adb3437f7d0fdb4f025051969" gracePeriod=30 Sep 10 15:16:43.699592 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.699559 2560 generic.go:356] "Generic (PLEG): container finished" podID="3256b1ff-16f6-453a-9476-7b61d5ec4ffb" containerID="8a6c28f3d371b5697e36c6d7b7592c5c9919337adb3437f7d0fdb4f025051969" exitCode=0 Sep 10 15:16:43.699768 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.699631 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" event={"ID":"3256b1ff-16f6-453a-9476-7b61d5ec4ffb","Type":"ContainerDied","Data":"8a6c28f3d371b5697e36c6d7b7592c5c9919337adb3437f7d0fdb4f025051969"} Sep 10 15:16:43.793652 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.793599 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:16:43.908814 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.908765 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-metrics-certs\") pod \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " Sep 10 15:16:43.908992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.908875 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-openshift-service-ca\") pod \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " Sep 10 15:16:43.908992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.908903 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls\") pod \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " Sep 10 15:16:43.908992 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.908934 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzxpz\" (UniqueName: \"kubernetes.io/projected/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-kube-api-access-fzxpz\") pod \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\" (UID: \"3256b1ff-16f6-453a-9476-7b61d5ec4ffb\") " Sep 10 15:16:43.909242 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.909218 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3256b1ff-16f6-453a-9476-7b61d5ec4ffb" (UID: "3256b1ff-16f6-453a-9476-7b61d5ec4ffb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:16:43.910944 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.910918 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-kube-api-access-fzxpz" (OuterVolumeSpecName: "kube-api-access-fzxpz") pod "3256b1ff-16f6-453a-9476-7b61d5ec4ffb" (UID: "3256b1ff-16f6-453a-9476-7b61d5ec4ffb"). InnerVolumeSpecName "kube-api-access-fzxpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:16:43.910944 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.910935 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3256b1ff-16f6-453a-9476-7b61d5ec4ffb" (UID: "3256b1ff-16f6-453a-9476-7b61d5ec4ffb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:16:43.911075 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:43.910920 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3256b1ff-16f6-453a-9476-7b61d5ec4ffb" (UID: "3256b1ff-16f6-453a-9476-7b61d5ec4ffb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:16:44.010101 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.010064 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:44.010101 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.010097 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:44.010101 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.010109 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:44.010334 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.010117 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxpz\" (UniqueName: \"kubernetes.io/projected/3256b1ff-16f6-453a-9476-7b61d5ec4ffb-kube-api-access-fzxpz\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:16:44.706071 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.706030 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" event={"ID":"3256b1ff-16f6-453a-9476-7b61d5ec4ffb","Type":"ContainerDied","Data":"c13a635efe03e2fe3b1d277e5ae1846e12a1ae71b52d8de6f0c354271ef037b5"} Sep 10 15:16:44.706479 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.706089 2560 scope.go:117] "RemoveContainer" containerID="8a6c28f3d371b5697e36c6d7b7592c5c9919337adb3437f7d0fdb4f025051969" Sep 10 15:16:44.706479 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.706041 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv" Sep 10 15:16:44.728652 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.728629 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv"] Sep 10 15:16:44.732695 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:44.732672 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13ee06-6cc69d6db-w9fbv"] Sep 10 15:16:45.665318 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:16:45.665284 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3256b1ff-16f6-453a-9476-7b61d5ec4ffb" path="/var/lib/kubelet/pods/3256b1ff-16f6-453a-9476-7b61d5ec4ffb/volumes" Sep 10 15:17:31.983472 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:31.983436 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f"] Sep 10 15:17:31.984085 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:31.983665 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" containerID="cri-o://c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe" gracePeriod=30 Sep 10 15:17:37.240903 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.240878 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:17:37.337377 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337284 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-token\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337377 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337320 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istiod-ca-cert\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337377 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337358 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-credential-socket\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337658 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337474 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-data\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337658 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337521 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-socket\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337658 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337605 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-certs\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337658 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337629 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:37.337658 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337654 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpzp5\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-kube-api-access-fpzp5\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337702 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-envoy\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337749 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-podinfo\") pod \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\" (UID: \"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0\") " Sep 10 15:17:37.337934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337829 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:17:37.337934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337824 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-data" (OuterVolumeSpecName: "istio-data") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:37.337934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337856 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:37.337934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.337871 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:37.338228 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.338085 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.338228 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.338105 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.338228 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.338118 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.338228 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.338126 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.338228 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.338134 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.339759 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.339734 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-kube-api-access-fpzp5" (OuterVolumeSpecName: "kube-api-access-fpzp5") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "kube-api-access-fpzp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:17:37.339888 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.339764 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:17:37.339971 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.339955 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-token" (OuterVolumeSpecName: "istio-token") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:17:37.340031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.340019 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" (UID: "1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:37.438726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.438676 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpzp5\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-kube-api-access-fpzp5\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.438726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.438710 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.438726 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.438724 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.438991 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.438739 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:37.972625 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.972532 2560 generic.go:356] "Generic (PLEG): container finished" podID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerID="c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe" exitCode=0 Sep 10 15:17:37.972625 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.972576 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" Sep 10 15:17:37.972884 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.972570 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" event={"ID":"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0","Type":"ContainerDied","Data":"c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe"} Sep 10 15:17:37.972884 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.972706 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f" event={"ID":"1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0","Type":"ContainerDied","Data":"37a749c9c7e15f989bb057efc6962146cf556d000bbcbdb39471601147a388b6"} Sep 10 15:17:37.972884 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.972738 2560 scope.go:117] "RemoveContainer" containerID="c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe" Sep 10 15:17:37.981712 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.981696 2560 scope.go:117] "RemoveContainer" containerID="0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde" Sep 10 15:17:37.989675 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.989653 2560 scope.go:117] "RemoveContainer" containerID="c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe" Sep 10 15:17:37.990210 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:17:37.990182 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe\": container with ID starting with c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe not found: ID does not exist" containerID="c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe" Sep 10 15:17:37.990321 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.990223 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe"} err="failed to get container status \"c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe\": rpc error: code = NotFound desc = could not find container \"c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe\": container with ID starting with c0af6379aa7dab9ab6c9673c6844b48f3e3029db3fd701c3a1576752897554fe not found: ID does not exist" Sep 10 15:17:37.990321 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.990266 2560 scope.go:117] "RemoveContainer" containerID="0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde" Sep 10 15:17:37.990840 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:17:37.990793 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde\": container with ID starting with 0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde not found: ID does not exist" containerID="0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde" Sep 10 15:17:37.990954 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.990844 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde"} err="failed to get container status \"0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde\": rpc error: code = NotFound desc = could not find container \"0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde\": container with ID starting with 0d483694c8e9f52562dc56c6d9254e129e3516a8dffdaba4dabcf04bbd833bde not found: ID does not exist" Sep 10 15:17:37.992484 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.992462 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f"] Sep 10 15:17:37.998005 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:37.997985 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-13ee06-openshift-default-84fd467f7d-sjs9f"] Sep 10 15:17:39.665449 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:39.665414 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" path="/var/lib/kubelet/pods/1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0/volumes" Sep 10 15:17:41.804599 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:41.804568 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:17:41.806560 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:41.806539 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:17:49.697278 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:49.697229 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm"] Sep 10 15:17:49.697879 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:49.697483 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" containerID="cri-o://b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258" gracePeriod=30 Sep 10 15:17:54.949386 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:54.949357 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:17:55.013364 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013093 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-podinfo\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013364 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013153 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-token\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013364 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013287 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140d701a-71b6-42d8-804c-4048eadbbdfa-istiod-ca-cert\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013364 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013315 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-credential-socket\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013364 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013360 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-socket\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013773 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013399 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-certs\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013773 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013426 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-data\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013773 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013449 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-envoy\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013773 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013475 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hmzz\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-kube-api-access-2hmzz\") pod \"140d701a-71b6-42d8-804c-4048eadbbdfa\" (UID: \"140d701a-71b6-42d8-804c-4048eadbbdfa\") " Sep 10 15:17:55.013773 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013725 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:55.013773 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013740 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:55.014080 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013847 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.014080 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013846 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140d701a-71b6-42d8-804c-4048eadbbdfa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:17:55.014080 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013865 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.014080 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.013986 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:55.014080 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.014068 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-data" (OuterVolumeSpecName: "istio-data") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:55.015770 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.015741 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:17:55.015930 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.015767 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-token" (OuterVolumeSpecName: "istio-token") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:17:55.015930 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.015766 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-kube-api-access-2hmzz" (OuterVolumeSpecName: "kube-api-access-2hmzz") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "kube-api-access-2hmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:17:55.016087 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.015960 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "140d701a-71b6-42d8-804c-4048eadbbdfa" (UID: "140d701a-71b6-42d8-804c-4048eadbbdfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:17:55.071311 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.071280 2560 generic.go:356] "Generic (PLEG): container finished" podID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerID="b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258" exitCode=0 Sep 10 15:17:55.071442 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.071326 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" Sep 10 15:17:55.071442 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.071372 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" event={"ID":"140d701a-71b6-42d8-804c-4048eadbbdfa","Type":"ContainerDied","Data":"b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258"} Sep 10 15:17:55.071442 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.071428 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm" event={"ID":"140d701a-71b6-42d8-804c-4048eadbbdfa","Type":"ContainerDied","Data":"c08a9a845be66be616cbb817197723ee18d0dc945256093c5f0c6f7a4a4fb4f0"} Sep 10 15:17:55.071546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.071451 2560 scope.go:117] "RemoveContainer" containerID="b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258" Sep 10 15:17:55.080542 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.080523 2560 scope.go:117] "RemoveContainer" containerID="3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3" Sep 10 15:17:55.088422 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.088405 2560 scope.go:117] "RemoveContainer" containerID="b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258" Sep 10 15:17:55.088692 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:17:55.088674 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258\": container with ID starting with b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258 not found: ID does not exist" containerID="b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258" Sep 10 15:17:55.088754 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.088700 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258"} err="failed to get container status \"b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258\": rpc error: code = NotFound desc = could not find container \"b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258\": container with ID starting with b0f5955ac06a7a49c236dd94e13371471603441546652475b548abea978df258 not found: ID does not exist" Sep 10 15:17:55.088754 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.088718 2560 scope.go:117] "RemoveContainer" containerID="3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3" Sep 10 15:17:55.088990 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:17:55.088970 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3\": container with ID starting with 3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3 not found: ID does not exist" containerID="3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3" Sep 10 15:17:55.089035 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.088996 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3"} err="failed to get container status \"3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3\": rpc error: code = NotFound desc = could not find container \"3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3\": container with ID starting with 3b4b2dd4f6b656e430b4621b0c5bc6300cdf928066cdc5e789b1371afaec49b3 not found: ID does not exist" Sep 10 15:17:55.093852 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.093831 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm"] Sep 10 15:17:55.100210 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.100188 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6333de-openshift-default-65b875fb68-lcfzm"] Sep 10 15:17:55.114938 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.114921 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140d701a-71b6-42d8-804c-4048eadbbdfa-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.114938 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.114940 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.115043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.114949 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.115043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.114958 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.115043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.114967 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hmzz\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-kube-api-access-2hmzz\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.115043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.114976 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.115043 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.114985 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140d701a-71b6-42d8-804c-4048eadbbdfa-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:17:55.664893 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:17:55.664865 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" path="/var/lib/kubelet/pods/140d701a-71b6-42d8-804c-4048eadbbdfa/volumes" Sep 10 15:18:07.205719 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:07.205683 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t"] Sep 10 15:18:07.206297 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:07.205963 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" containerID="cri-o://71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0" gracePeriod=30 Sep 10 15:18:12.452704 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.452680 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:18:12.582918 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.582838 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-data\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.582918 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.582899 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-credential-socket\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.583136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.582950 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-socket\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.583136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.582969 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-envoy\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.583136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.582996 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-podinfo\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.583136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583036 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-certs\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.583136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583055 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-token\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.583136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583077 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istiod-ca-cert\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.583136 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583118 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q47ln\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-kube-api-access-q47ln\") pod \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\" (UID: \"3a935cb7-ecc4-42f3-9937-7b5ccf35e929\") " Sep 10 15:18:12.583517 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583140 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:12.583517 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583218 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:12.583517 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583230 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:12.583517 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583411 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:12.583732 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583583 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:12.583732 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583607 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:12.583732 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583616 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:12.583732 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583623 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:12.583732 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.583579 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:18:12.585331 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.585305 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:12.585331 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.585331 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:12.585486 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.585343 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-kube-api-access-q47ln" (OuterVolumeSpecName: "kube-api-access-q47ln") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "kube-api-access-q47ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:12.585486 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.585349 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a935cb7-ecc4-42f3-9937-7b5ccf35e929" (UID: "3a935cb7-ecc4-42f3-9937-7b5ccf35e929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:18:12.684424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.684390 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:12.684424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.684416 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:12.684424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.684425 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:12.684424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.684433 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:12.684676 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:12.684441 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47ln\" (UniqueName: \"kubernetes.io/projected/3a935cb7-ecc4-42f3-9937-7b5ccf35e929-kube-api-access-q47ln\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:13.161034 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.160990 2560 generic.go:356] "Generic (PLEG): container finished" podID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerID="71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0" exitCode=0 Sep 10 15:18:13.161034 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.161036 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" Sep 10 15:18:13.161249 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.161084 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" event={"ID":"3a935cb7-ecc4-42f3-9937-7b5ccf35e929","Type":"ContainerDied","Data":"71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0"} Sep 10 15:18:13.161249 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.161134 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t" event={"ID":"3a935cb7-ecc4-42f3-9937-7b5ccf35e929","Type":"ContainerDied","Data":"55fbda06a08067d70b91e625f8f3b782d959115e8f99d9972b0b6dbece5722e1"} Sep 10 15:18:13.161249 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.161156 2560 scope.go:117] "RemoveContainer" containerID="71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0" Sep 10 15:18:13.170710 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.170689 2560 scope.go:117] "RemoveContainer" containerID="ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea" Sep 10 15:18:13.178351 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.178331 2560 scope.go:117] "RemoveContainer" containerID="71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0" Sep 10 15:18:13.178616 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:18:13.178597 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0\": container with ID starting with 71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0 not found: ID does not exist" containerID="71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0" Sep 10 15:18:13.178658 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.178625 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0"} err="failed to get container status \"71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0\": rpc error: code = NotFound desc = could not find container \"71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0\": container with ID starting with 71dcf2bbb4fbefcb49311361bc09b1be4468227b474201d8fd3183bd1182a8e0 not found: ID does not exist" Sep 10 15:18:13.178658 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.178642 2560 scope.go:117] "RemoveContainer" containerID="ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea" Sep 10 15:18:13.178919 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:18:13.178900 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea\": container with ID starting with ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea not found: ID does not exist" containerID="ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea" Sep 10 15:18:13.178965 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.178927 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea"} err="failed to get container status \"ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea\": rpc error: code = NotFound desc = could not find container \"ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea\": container with ID starting with ed4e8012af47919282294c18e21efbaf8c336c4800e9b0acdf8591187d27baea not found: ID does not exist" Sep 10 15:18:13.186836 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.186514 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t"] Sep 10 15:18:13.193016 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.192992 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-decc52-openshift-default-66645b8b5b-mwn2t"] Sep 10 15:18:13.664671 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:13.664636 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" path="/var/lib/kubelet/pods/3a935cb7-ecc4-42f3-9937-7b5ccf35e929/volumes" Sep 10 15:18:18.217083 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.217052 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw"] Sep 10 15:18:18.217582 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.217283 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" podUID="3f241998-59ca-4fb4-91fb-59a2dc7c37c5" containerName="maas-api" containerID="cri-o://dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e" gracePeriod=30 Sep 10 15:18:18.462854 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.462830 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:18:18.640607 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.640569 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-metrics-certs\") pod \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " Sep 10 15:18:18.640776 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.640759 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls\") pod \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " Sep 10 15:18:18.640873 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.640857 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9mdk\" (UniqueName: \"kubernetes.io/projected/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-kube-api-access-r9mdk\") pod \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " Sep 10 15:18:18.640922 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.640905 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-openshift-service-ca\") pod \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\" (UID: \"3f241998-59ca-4fb4-91fb-59a2dc7c37c5\") " Sep 10 15:18:18.641349 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.641318 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f241998-59ca-4fb4-91fb-59a2dc7c37c5" (UID: "3f241998-59ca-4fb4-91fb-59a2dc7c37c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:18:18.642815 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.642779 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f241998-59ca-4fb4-91fb-59a2dc7c37c5" (UID: "3f241998-59ca-4fb4-91fb-59a2dc7c37c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:18:18.642934 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.642910 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-kube-api-access-r9mdk" (OuterVolumeSpecName: "kube-api-access-r9mdk") pod "3f241998-59ca-4fb4-91fb-59a2dc7c37c5" (UID: "3f241998-59ca-4fb4-91fb-59a2dc7c37c5"). InnerVolumeSpecName "kube-api-access-r9mdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:18.643009 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.642993 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f241998-59ca-4fb4-91fb-59a2dc7c37c5" (UID: "3f241998-59ca-4fb4-91fb-59a2dc7c37c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:18:18.742200 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.742162 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:18.742200 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.742194 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9mdk\" (UniqueName: \"kubernetes.io/projected/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-kube-api-access-r9mdk\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:18.742200 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.742204 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:18.742432 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:18.742213 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f241998-59ca-4fb4-91fb-59a2dc7c37c5-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:19.191557 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.191520 2560 generic.go:356] "Generic (PLEG): container finished" podID="3f241998-59ca-4fb4-91fb-59a2dc7c37c5" containerID="dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e" exitCode=0 Sep 10 15:18:19.191725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.191570 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" Sep 10 15:18:19.191725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.191607 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" event={"ID":"3f241998-59ca-4fb4-91fb-59a2dc7c37c5","Type":"ContainerDied","Data":"dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e"} Sep 10 15:18:19.191725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.191663 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw" event={"ID":"3f241998-59ca-4fb4-91fb-59a2dc7c37c5","Type":"ContainerDied","Data":"1a1f2270b6b3b71dab84697dd33d60f29545a01dd74c3205cb9446bc8d03b4bd"} Sep 10 15:18:19.191725 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.191685 2560 scope.go:117] "RemoveContainer" containerID="dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e" Sep 10 15:18:19.201113 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.201097 2560 scope.go:117] "RemoveContainer" containerID="dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e" Sep 10 15:18:19.201387 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:18:19.201365 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e\": container with ID starting with dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e not found: ID does not exist" containerID="dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e" Sep 10 15:18:19.201467 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.201399 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e"} err="failed to get container status \"dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e\": rpc error: code = NotFound desc = could not find container \"dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e\": container with ID starting with dc89dfeefcb8ca0960bd09c77f025f1fa5173e766dffcccc34903a3c6030995e not found: ID does not exist" Sep 10 15:18:19.213077 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.213051 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw"] Sep 10 15:18:19.214728 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.214708 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5a2df-5dff676f74-lcrsw"] Sep 10 15:18:19.664299 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:19.664268 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f241998-59ca-4fb4-91fb-59a2dc7c37c5" path="/var/lib/kubelet/pods/3f241998-59ca-4fb4-91fb-59a2dc7c37c5/volumes" Sep 10 15:18:25.127952 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:25.127917 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn"] Sep 10 15:18:25.128391 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:25.128193 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" containerID="cri-o://80612fe929c7afcd2dc4e1ad8a40d4979a71b38d97248c96d333d5d8a971d270" gracePeriod=30 Sep 10 15:18:30.246638 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.246607 2560 generic.go:356] "Generic (PLEG): container finished" podID="6efce121-55ac-4c92-8717-88557ae30e1a" containerID="80612fe929c7afcd2dc4e1ad8a40d4979a71b38d97248c96d333d5d8a971d270" exitCode=0 Sep 10 15:18:30.246967 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.246692 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" event={"ID":"6efce121-55ac-4c92-8717-88557ae30e1a","Type":"ContainerDied","Data":"80612fe929c7afcd2dc4e1ad8a40d4979a71b38d97248c96d333d5d8a971d270"} Sep 10 15:18:30.246967 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.246775 2560 scope.go:117] "RemoveContainer" containerID="b4abc51a2d514a4d1d5fcb4ad3b75b27b8a6ec6e109da6f087374652017010fa" Sep 10 15:18:30.397790 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.397729 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:18:30.440871 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.440841 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-socket\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.440904 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-envoy\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.440922 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efce121-55ac-4c92-8717-88557ae30e1a-istiod-ca-cert\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.440942 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-istio-token\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.440963 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-credential-socket\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.440993 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-data\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441031 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441015 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efce121-55ac-4c92-8717-88557ae30e1a-istio-podinfo\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441049 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-certs\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441078 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfq8f\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-kube-api-access-wfq8f\") pod \"6efce121-55ac-4c92-8717-88557ae30e1a\" (UID: \"6efce121-55ac-4c92-8717-88557ae30e1a\") " Sep 10 15:18:30.441372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441120 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:30.441372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441255 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:30.441372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441327 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:30.441372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441331 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6efce121-55ac-4c92-8717-88557ae30e1a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:18:30.441372 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441344 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-credential-socket\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:30.441747 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441500 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:30.441747 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.441591 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-data" (OuterVolumeSpecName: "istio-data") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:30.443635 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.443601 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-kube-api-access-wfq8f" (OuterVolumeSpecName: "kube-api-access-wfq8f") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "kube-api-access-wfq8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:30.443745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.443653 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6efce121-55ac-4c92-8717-88557ae30e1a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:18:30.443745 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.443663 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:18:30.443878 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.443862 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-istio-token" (OuterVolumeSpecName: "istio-token") pod "6efce121-55ac-4c92-8717-88557ae30e1a" (UID: "6efce121-55ac-4c92-8717-88557ae30e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:18:30.542494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.542454 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-envoy\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:30.542494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.542489 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efce121-55ac-4c92-8717-88557ae30e1a-istiod-ca-cert\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:30.542494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.542498 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-istio-token\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:30.542494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.542506 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-istio-data\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:30.542723 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.542514 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efce121-55ac-4c92-8717-88557ae30e1a-istio-podinfo\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:30.542723 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.542521 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efce121-55ac-4c92-8717-88557ae30e1a-workload-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:30.542723 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:30.542528 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfq8f\" (UniqueName: \"kubernetes.io/projected/6efce121-55ac-4c92-8717-88557ae30e1a-kube-api-access-wfq8f\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:18:31.253267 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:31.253231 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" event={"ID":"6efce121-55ac-4c92-8717-88557ae30e1a","Type":"ContainerDied","Data":"4ee43301e6ffca8cdb756f87f3c8d16328e848b3875077e74f37815eee7ddffd"} Sep 10 15:18:31.253267 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:31.253253 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn" Sep 10 15:18:31.253768 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:31.253296 2560 scope.go:117] "RemoveContainer" containerID="80612fe929c7afcd2dc4e1ad8a40d4979a71b38d97248c96d333d5d8a971d270" Sep 10 15:18:31.273748 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:31.273723 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn"] Sep 10 15:18:31.277544 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:31.277522 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d5a2df-openshift-default-847d77686-9qkjn"] Sep 10 15:18:31.664403 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:18:31.664372 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" path="/var/lib/kubelet/pods/6efce121-55ac-4c92-8717-88557ae30e1a/volumes" Sep 10 15:22:41.837099 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:22:41.837070 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:22:41.841318 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:22:41.841292 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-92lh7_9ccb0608-8ad0-48d7-bb71-3f258e830b99/ovn-acl-logging/0.log" Sep 10 15:24:40.349361 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349316 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2"] Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349753 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349767 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349777 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349783 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349793 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3256b1ff-16f6-453a-9476-7b61d5ec4ffb" containerName="maas-api" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349811 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3256b1ff-16f6-453a-9476-7b61d5ec4ffb" containerName="maas-api" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349821 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349827 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349835 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349841 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349849 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f241998-59ca-4fb4-91fb-59a2dc7c37c5" containerName="maas-api" Sep 10 15:24:40.349848 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349855 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f241998-59ca-4fb4-91fb-59a2dc7c37c5" containerName="maas-api" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349861 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349866 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349873 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349878 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349887 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349892 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349906 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349912 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349918 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349923 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349929 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349934 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349990 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f241998-59ca-4fb4-91fb-59a2dc7c37c5" containerName="maas-api" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.349998 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.350003 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.350009 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.350015 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.350021 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="140d701a-71b6-42d8-804c-4048eadbbdfa" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.350027 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.350034 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6efce121-55ac-4c92-8717-88557ae30e1a" containerName="istio-proxy" Sep 10 15:24:40.350234 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.350047 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3256b1ff-16f6-453a-9476-7b61d5ec4ffb" containerName="maas-api" Sep 10 15:24:40.353267 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.353236 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.377917 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.377876 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2"] Sep 10 15:24:40.396462 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.396412 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680498eb-e25d-4556-8229-ccdbd6da1557-maas-api-tls\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.396713 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.396474 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680498eb-e25d-4556-8229-ccdbd6da1557-openshift-service-ca\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.396713 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.396510 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4mg4\" (UniqueName: \"kubernetes.io/projected/680498eb-e25d-4556-8229-ccdbd6da1557-kube-api-access-j4mg4\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.396713 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.396596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680498eb-e25d-4556-8229-ccdbd6da1557-metrics-certs\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.498161 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.498113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680498eb-e25d-4556-8229-ccdbd6da1557-maas-api-tls\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.498161 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.498164 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680498eb-e25d-4556-8229-ccdbd6da1557-openshift-service-ca\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.498494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.498186 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4mg4\" (UniqueName: \"kubernetes.io/projected/680498eb-e25d-4556-8229-ccdbd6da1557-kube-api-access-j4mg4\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.498494 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.498215 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680498eb-e25d-4556-8229-ccdbd6da1557-metrics-certs\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.499022 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.498994 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680498eb-e25d-4556-8229-ccdbd6da1557-openshift-service-ca\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.501208 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.501173 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680498eb-e25d-4556-8229-ccdbd6da1557-metrics-certs\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.501326 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.501220 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680498eb-e25d-4556-8229-ccdbd6da1557-maas-api-tls\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.507284 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.507247 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4mg4\" (UniqueName: \"kubernetes.io/projected/680498eb-e25d-4556-8229-ccdbd6da1557-kube-api-access-j4mg4\") pod \"maas-api-5cc85db8b4-bxzl2\" (UID: \"680498eb-e25d-4556-8229-ccdbd6da1557\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.664988 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.664924 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:40.941784 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.941759 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2"] Sep 10 15:24:40.943901 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:24:40.943873 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680498eb_e25d_4556_8229_ccdbd6da1557.slice/crio-ed7b0591d5a10fc891d7602715812d01aff972737ed2f07253b25ace4d535ae8 WatchSource:0}: Error finding container ed7b0591d5a10fc891d7602715812d01aff972737ed2f07253b25ace4d535ae8: Status 404 returned error can't find the container with id ed7b0591d5a10fc891d7602715812d01aff972737ed2f07253b25ace4d535ae8 Sep 10 15:24:40.945266 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:40.945244 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:24:41.035126 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:41.035088 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" event={"ID":"680498eb-e25d-4556-8229-ccdbd6da1557","Type":"ContainerStarted","Data":"ed7b0591d5a10fc891d7602715812d01aff972737ed2f07253b25ace4d535ae8"} Sep 10 15:24:42.041710 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:42.041673 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" event={"ID":"680498eb-e25d-4556-8229-ccdbd6da1557","Type":"ContainerStarted","Data":"ab20d5534b3f4086689a156856fd2918293951cc277a8a59bc9b4273459e3c1f"} Sep 10 15:24:42.042163 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:42.041790 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:48.052176 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:48.052145 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" Sep 10 15:24:48.069997 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:48.069942 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-bxzl2" podStartSLOduration=7.568552169 podStartE2EDuration="8.069923978s" totalImagesPullingTime="501.371809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:24:42.058859342 +0000 UTC m=+2220.873442095" watchObservedRunningTime="2026-09-10 15:24:48.069923978 +0000 UTC m=+2226.884506729" Sep 10 15:24:48.097771 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:48.097734 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt"] Sep 10 15:24:48.098050 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:48.098017 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" podUID="5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" containerName="maas-api" containerID="cri-o://23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708" gracePeriod=30 Sep 10 15:24:49.363744 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.363711 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 15:24:49.491888 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.491776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzcw8\" (UniqueName: \"kubernetes.io/projected/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-kube-api-access-wzcw8\") pod \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " Sep 10 15:24:49.492096 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.491887 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-metrics-certs\") pod \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " Sep 10 15:24:49.492096 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.491982 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-openshift-service-ca\") pod \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " Sep 10 15:24:49.492096 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.492017 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-maas-api-tls\") pod \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\" (UID: \"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20\") " Sep 10 15:24:49.492639 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.492465 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" (UID: "5b0f3cc0-ff2f-40ee-970f-baa8a11eac20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:24:49.494424 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.494388 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" (UID: "5b0f3cc0-ff2f-40ee-970f-baa8a11eac20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:24:49.494643 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.494620 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" (UID: "5b0f3cc0-ff2f-40ee-970f-baa8a11eac20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:24:49.494715 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.494649 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-kube-api-access-wzcw8" (OuterVolumeSpecName: "kube-api-access-wzcw8") pod "5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" (UID: "5b0f3cc0-ff2f-40ee-970f-baa8a11eac20"). InnerVolumeSpecName "kube-api-access-wzcw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:24:49.593145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.593110 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-metrics-certs\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:24:49.593145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.593140 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-openshift-service-ca\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:24:49.593145 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.593151 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-maas-api-tls\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:24:49.593405 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:49.593160 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzcw8\" (UniqueName: \"kubernetes.io/projected/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20-kube-api-access-wzcw8\") on node \"ip-10-0-143-4.ec2.internal\" DevicePath \"\"" Sep 10 15:24:50.085007 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.084966 2560 generic.go:356] "Generic (PLEG): container finished" podID="5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" containerID="23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708" exitCode=0 Sep 10 15:24:50.085242 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.085014 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" Sep 10 15:24:50.085242 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.085060 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" event={"ID":"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20","Type":"ContainerDied","Data":"23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708"} Sep 10 15:24:50.085242 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.085122 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt" event={"ID":"5b0f3cc0-ff2f-40ee-970f-baa8a11eac20","Type":"ContainerDied","Data":"45f542690eec7ac6a927bead6229ab166407caba99aca887b16b28639a3bbba2"} Sep 10 15:24:50.085242 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.085153 2560 scope.go:117] "RemoveContainer" containerID="23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708" Sep 10 15:24:50.095056 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.095032 2560 scope.go:117] "RemoveContainer" containerID="23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708" Sep 10 15:24:50.095431 ip-10-0-143-4 kubenswrapper[2560]: E0910 15:24:50.095406 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708\": container with ID starting with 23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708 not found: ID does not exist" containerID="23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708" Sep 10 15:24:50.095545 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.095440 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708"} err="failed to get container status \"23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708\": rpc error: code = NotFound desc = could not find container \"23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708\": container with ID starting with 23bf85cb42ac62fc5fd1631e4f5066ad91f4816dc0dcda0adc1056414cdd7708 not found: ID does not exist" Sep 10 15:24:50.101724 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.101682 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt"] Sep 10 15:24:50.106297 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:50.106260 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-gxmkt"] Sep 10 15:24:51.666068 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:24:51.666029 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" path="/var/lib/kubelet/pods/5b0f3cc0-ff2f-40ee-970f-baa8a11eac20/volumes" Sep 10 15:25:15.477396 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:15.477315 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v7dgr_8c96ead6-6b3b-4286-a355-4adfc1e8c78f/server/0.log" Sep 10 15:25:15.610748 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:15.610713 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dtqvt_03c2fc40-42f4-4121-836f-2a72f9c96a08/manager/0.log" Sep 10 15:25:16.004354 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:16.004325 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-x6spr_04df52ba-2f52-4ea1-9b82-9b562cc362c1/manager/0.log" Sep 10 15:25:16.010317 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:16.010297 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-x6spr_04df52ba-2f52-4ea1-9b82-9b562cc362c1/copy-manifests/0.log" Sep 10 15:25:16.338902 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:16.338822 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cc85db8b4-bxzl2_680498eb-e25d-4556-8229-ccdbd6da1557/maas-api/0.log" Sep 10 15:25:18.523612 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:18.523580 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf_d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4/istio-proxy/0.log" Sep 10 15:25:18.641067 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:18.641035 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j8r5r_708f4997-9623-4ba9-b6fa-0471141288e7/discovery/0.log" Sep 10 15:25:18.982466 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:18.982432 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-t22tb_39c1404d-585b-46bf-98b8-7ca1985a600d/istio-proxy/1.log" Sep 10 15:25:19.764922 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:19.764888 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-rcbds_fa1b823d-26d0-444d-9eb0-8f960673787e/main/0.log" Sep 10 15:25:28.790931 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:28.790904 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d2txf_549f4770-f5a2-49d8-a3ce-d62df642b6e7/global-pull-secret-syncer/0.log" Sep 10 15:25:29.015957 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:29.015921 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p87s2_ed237cec-68e7-468d-ba17-ac16ced33017/konnectivity-agent/0.log" Sep 10 15:25:29.093732 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:29.093656 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-4.ec2.internal_b2320f504613112cbb2f124197a41f0d/haproxy/0.log" Sep 10 15:25:35.361876 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.361844 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91e1401f-0ccf-4b5c-85e3-672f09a45620/alertmanager/0.log" Sep 10 15:25:35.385135 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.385067 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91e1401f-0ccf-4b5c-85e3-672f09a45620/config-reloader/0.log" Sep 10 15:25:35.407864 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.407840 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91e1401f-0ccf-4b5c-85e3-672f09a45620/kube-rbac-proxy-web/0.log" Sep 10 15:25:35.436044 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.436023 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91e1401f-0ccf-4b5c-85e3-672f09a45620/kube-rbac-proxy/0.log" Sep 10 15:25:35.459807 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.459782 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91e1401f-0ccf-4b5c-85e3-672f09a45620/kube-rbac-proxy-metric/0.log" Sep 10 15:25:35.487953 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.487926 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91e1401f-0ccf-4b5c-85e3-672f09a45620/prom-label-proxy/0.log" Sep 10 15:25:35.518772 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.518748 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91e1401f-0ccf-4b5c-85e3-672f09a45620/init-config-reloader/0.log" Sep 10 15:25:35.605582 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.605561 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bpcgr_19e46b0f-b5bd-44fb-aa8f-ae38f5555e01/kube-state-metrics/0.log" Sep 10 15:25:35.633719 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.633698 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bpcgr_19e46b0f-b5bd-44fb-aa8f-ae38f5555e01/kube-rbac-proxy-main/0.log" Sep 10 15:25:35.661602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.661529 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-bpcgr_19e46b0f-b5bd-44fb-aa8f-ae38f5555e01/kube-rbac-proxy-self/0.log" Sep 10 15:25:35.723856 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:35.723833 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-s9n4d_41904cb2-4d32-486c-bf85-b6c601e61b12/monitoring-plugin/0.log" Sep 10 15:25:36.012313 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.012232 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfnbw_3db175b6-6536-4b9c-854d-0630040d4d33/node-exporter/0.log" Sep 10 15:25:36.038204 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.038182 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfnbw_3db175b6-6536-4b9c-854d-0630040d4d33/kube-rbac-proxy/0.log" Sep 10 15:25:36.064912 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.064894 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfnbw_3db175b6-6536-4b9c-854d-0630040d4d33/init-textfile/0.log" Sep 10 15:25:36.224390 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.224361 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cc3da546-573b-48e7-996a-23d87c252b44/prometheus/0.log" Sep 10 15:25:36.256169 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.256143 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cc3da546-573b-48e7-996a-23d87c252b44/config-reloader/0.log" Sep 10 15:25:36.284247 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.284191 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cc3da546-573b-48e7-996a-23d87c252b44/thanos-sidecar/0.log" Sep 10 15:25:36.309744 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.309724 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cc3da546-573b-48e7-996a-23d87c252b44/kube-rbac-proxy-web/0.log" Sep 10 15:25:36.333647 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.333623 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cc3da546-573b-48e7-996a-23d87c252b44/kube-rbac-proxy/0.log" Sep 10 15:25:36.362986 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.362960 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cc3da546-573b-48e7-996a-23d87c252b44/kube-rbac-proxy-thanos/0.log" Sep 10 15:25:36.393498 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.393475 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cc3da546-573b-48e7-996a-23d87c252b44/init-config-reloader/0.log" Sep 10 15:25:36.497930 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.497904 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-jmzff_087e86a0-662f-4251-a2d4-33d14e401415/prometheus-operator-admission-webhook/0.log" Sep 10 15:25:36.688888 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.688826 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-764bd5c656-xb7f4_3fa1eb81-b11f-4bc0-8794-9b05406dbc23/thanos-query/0.log" Sep 10 15:25:36.722389 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.722366 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-764bd5c656-xb7f4_3fa1eb81-b11f-4bc0-8794-9b05406dbc23/kube-rbac-proxy-web/0.log" Sep 10 15:25:36.748829 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.748789 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-764bd5c656-xb7f4_3fa1eb81-b11f-4bc0-8794-9b05406dbc23/kube-rbac-proxy/0.log" Sep 10 15:25:36.771752 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.771726 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-764bd5c656-xb7f4_3fa1eb81-b11f-4bc0-8794-9b05406dbc23/prom-label-proxy/0.log" Sep 10 15:25:36.794953 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.794935 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-764bd5c656-xb7f4_3fa1eb81-b11f-4bc0-8794-9b05406dbc23/kube-rbac-proxy-rules/0.log" Sep 10 15:25:36.818975 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:36.818953 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-764bd5c656-xb7f4_3fa1eb81-b11f-4bc0-8794-9b05406dbc23/kube-rbac-proxy-metrics/0.log" Sep 10 15:25:38.977874 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.977843 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk"] Sep 10 15:25:38.978248 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.978187 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" containerName="maas-api" Sep 10 15:25:38.978248 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.978198 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" containerName="maas-api" Sep 10 15:25:38.978324 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.978270 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd1d987-ac83-4ef7-9da9-454aa9a1aeb0" containerName="istio-proxy" Sep 10 15:25:38.978324 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.978279 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a935cb7-ecc4-42f3-9937-7b5ccf35e929" containerName="istio-proxy" Sep 10 15:25:38.978324 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.978286 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0f3cc0-ff2f-40ee-970f-baa8a11eac20" containerName="maas-api" Sep 10 15:25:38.978324 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.978292 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b793d983-6a7d-42fd-9451-fe05e139bd66" containerName="istio-proxy" Sep 10 15:25:38.987693 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.987656 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:38.990445 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.990417 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h5vch\"/\"openshift-service-ca.crt\"" Sep 10 15:25:38.990580 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.990432 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h5vch\"/\"kube-root-ca.crt\"" Sep 10 15:25:38.991046 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.991027 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h5vch\"/\"default-dockercfg-x2924\"" Sep 10 15:25:38.991749 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:38.991710 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk"] Sep 10 15:25:39.163174 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.163136 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-lib-modules\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.163349 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.163211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-proc\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.163349 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.163269 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94wc\" (UniqueName: \"kubernetes.io/projected/0219809b-61a1-4271-bc7e-d9cd7b274fd1-kube-api-access-l94wc\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.163349 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.163293 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-sys\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.163349 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.163323 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-podres\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.264534 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264435 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-proc\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.264719 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264580 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94wc\" (UniqueName: \"kubernetes.io/projected/0219809b-61a1-4271-bc7e-d9cd7b274fd1-kube-api-access-l94wc\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.264719 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-proc\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.264719 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264603 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-sys\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.264719 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264665 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-sys\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.264959 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264721 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-podres\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.264959 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264778 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-lib-modules\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.264959 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264931 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-podres\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.265075 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.264957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0219809b-61a1-4271-bc7e-d9cd7b274fd1-lib-modules\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.273824 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.273785 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94wc\" (UniqueName: \"kubernetes.io/projected/0219809b-61a1-4271-bc7e-d9cd7b274fd1-kube-api-access-l94wc\") pod \"perf-node-gather-daemonset-dw4mk\" (UID: \"0219809b-61a1-4271-bc7e-d9cd7b274fd1\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.298755 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.298728 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:39.569813 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:39.569772 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk"] Sep 10 15:25:39.572064 ip-10-0-143-4 kubenswrapper[2560]: W0910 15:25:39.572034 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0219809b_61a1_4271_bc7e_d9cd7b274fd1.slice/crio-f6f6e31cc3beb4cfadd9f4ba473d30bb52af4fa486ce6bfff088b7898a20a260 WatchSource:0}: Error finding container f6f6e31cc3beb4cfadd9f4ba473d30bb52af4fa486ce6bfff088b7898a20a260: Status 404 returned error can't find the container with id f6f6e31cc3beb4cfadd9f4ba473d30bb52af4fa486ce6bfff088b7898a20a260 Sep 10 15:25:40.325174 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:40.325143 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" event={"ID":"0219809b-61a1-4271-bc7e-d9cd7b274fd1","Type":"ContainerStarted","Data":"6da1c1968d21cb4e85de74de2acb5917af9de2f82367f7eed585708f8ea231bb"} Sep 10 15:25:40.325580 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:40.325190 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" event={"ID":"0219809b-61a1-4271-bc7e-d9cd7b274fd1","Type":"ContainerStarted","Data":"f6f6e31cc3beb4cfadd9f4ba473d30bb52af4fa486ce6bfff088b7898a20a260"} Sep 10 15:25:40.325580 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:40.325331 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:40.355194 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:40.355139 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" podStartSLOduration=2.355123207 podStartE2EDuration="2.355123207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:25:40.351490385 +0000 UTC m=+2279.166073148" watchObservedRunningTime="2026-09-10 15:25:40.355123207 +0000 UTC m=+2279.169705958" Sep 10 15:25:41.259063 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:41.259025 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4n624_4629d531-0365-416d-8177-92814deff28f/dns/0.log" Sep 10 15:25:41.279385 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:41.279363 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4n624_4629d531-0365-416d-8177-92814deff28f/kube-rbac-proxy/0.log" Sep 10 15:25:41.403075 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:41.403048 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4r7d6_20b78c3e-ce14-498b-bdbf-8baa9718867f/dns-node-resolver/0.log" Sep 10 15:25:42.080057 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:42.080025 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l298v_a8eb91f7-476a-4c3a-b163-adac5bc16180/node-ca/0.log" Sep 10 15:25:43.033655 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:43.033621 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf_d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4/istio-proxy/0.log" Sep 10 15:25:43.451275 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:43.451245 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j8r5r_708f4997-9623-4ba9-b6fa-0471141288e7/discovery/0.log" Sep 10 15:25:43.529371 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:43.529342 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-t22tb_39c1404d-585b-46bf-98b8-7ca1985a600d/istio-proxy/0.log" Sep 10 15:25:43.543912 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:43.543885 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-t22tb_39c1404d-585b-46bf-98b8-7ca1985a600d/istio-proxy/1.log" Sep 10 15:25:44.498735 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:44.498703 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8d5st_b4a99041-6bb6-472c-82d2-184121ef0adf/serve-healthcheck-canary/0.log" Sep 10 15:25:45.111846 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:45.111791 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bcfb2_b59b8e13-91dd-4cfd-9a36-52b88b999078/kube-rbac-proxy/0.log" Sep 10 15:25:45.133741 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:45.133720 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bcfb2_b59b8e13-91dd-4cfd-9a36-52b88b999078/exporter/0.log" Sep 10 15:25:45.155945 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:45.155923 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bcfb2_b59b8e13-91dd-4cfd-9a36-52b88b999078/extractor/0.log" Sep 10 15:25:46.340099 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:46.340073 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-dw4mk" Sep 10 15:25:47.988963 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:47.988938 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v7dgr_8c96ead6-6b3b-4286-a355-4adfc1e8c78f/server/0.log" Sep 10 15:25:48.122070 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:48.122041 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dtqvt_03c2fc40-42f4-4121-836f-2a72f9c96a08/manager/0.log" Sep 10 15:25:48.336587 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:48.336554 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-x6spr_04df52ba-2f52-4ea1-9b82-9b562cc362c1/manager/0.log" Sep 10 15:25:48.358878 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:48.358857 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-x6spr_04df52ba-2f52-4ea1-9b82-9b562cc362c1/copy-manifests/0.log" Sep 10 15:25:49.748546 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:49.748515 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6469457db4-kglvf_1d7cfe96-42a5-421c-b491-9c0d7cc54f59/manager/0.log" Sep 10 15:25:50.120093 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:50.120063 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf_d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4/istio-proxy/0.log" Sep 10 15:25:50.540100 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:50.540070 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j8r5r_708f4997-9623-4ba9-b6fa-0471141288e7/discovery/0.log" Sep 10 15:25:50.588154 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:50.588125 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-t22tb_39c1404d-585b-46bf-98b8-7ca1985a600d/istio-proxy/0.log" Sep 10 15:25:50.603967 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:50.603894 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-t22tb_39c1404d-585b-46bf-98b8-7ca1985a600d/istio-proxy/1.log" Sep 10 15:25:52.149997 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:52.149966 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ctmpzf_d09fbb8b-4f13-4542-bae5-7f0a4d7f7ba4/istio-proxy/0.log" Sep 10 15:25:52.587243 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:52.587167 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j8r5r_708f4997-9623-4ba9-b6fa-0471141288e7/discovery/0.log" Sep 10 15:25:52.641073 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:52.641034 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-t22tb_39c1404d-585b-46bf-98b8-7ca1985a600d/istio-proxy/0.log" Sep 10 15:25:52.658416 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:52.658350 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-t22tb_39c1404d-585b-46bf-98b8-7ca1985a600d/istio-proxy/1.log" Sep 10 15:25:54.865111 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:54.865077 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-c79tx_297067da-9436-4bae-9040-dd7911d76094/migrator/0.log" Sep 10 15:25:54.887088 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:54.887061 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-c79tx_297067da-9436-4bae-9040-dd7911d76094/graceful-termination/0.log" Sep 10 15:25:56.221247 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:56.221221 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56k7w_744583dc-6c66-45ed-9532-cd6950609908/kube-multus-additional-cni-plugins/0.log" Sep 10 15:25:56.250918 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:56.250889 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56k7w_744583dc-6c66-45ed-9532-cd6950609908/egress-router-binary-copy/0.log" Sep 10 15:25:56.276243 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:56.276219 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56k7w_744583dc-6c66-45ed-9532-cd6950609908/cni-plugins/0.log" Sep 10 15:25:56.308017 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:56.307996 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56k7w_744583dc-6c66-45ed-9532-cd6950609908/bond-cni-plugin/0.log" Sep 10 15:25:56.337602 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:56.337581 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56k7w_744583dc-6c66-45ed-9532-cd6950609908/routeoverride-cni/0.log" Sep 10 15:25:56.364643 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:56.364623 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56k7w_744583dc-6c66-45ed-9532-cd6950609908/whereabouts-cni-bincopy/0.log" Sep 10 15:25:56.392743 ip-10-0-143-4 kubenswrapper[2560]: I0910 15:25:56.392715 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56k7w_744583dc-6c66-45ed-9532-cd6950609908/whereabouts-cni/0.log"